RAND 


NATIONAL  DEFENSE 
RESEARCH  INSTITUTE 


CHILDREN  AND  FAMILIES 

The  RAND  Corporation  is  a  nonprofit  institution  that  helps  improve  policy  and 

EDUCATION  AND  THE  ARTS 

decisionmaking  through  research  and  analysis. 

ENERGY  AND  ENVIRONMENT 

HEALTH  AND  HEALTH  CARE 

This  electronic  document  was  made  available  from  www.rand.org  as  a  public  service 

INFRASTRUCTURE  AND 

TRANSPORTATION 

of  the  RAND  Corporation. 

INTERNATIONAL  AFFAIRS 

LAW  AND  BUSINESS 

Skip  all  front  matter:  lump  to  Page  1  ▼ 

NATIONAL  SECURITY 

POPULATION  AND  AGING 

PUBLIC  SAFETY 

Support  RAND 

SCIENCE  AND  TECHNOLOGY 

Purchase  this  document 

TERRORISM  AND 

HOMELAND  SECURITY 

Browse  Reports  &  Bookstore 

Make  a  charitable  contribution 

For  More  Information 

Visit  RAND  at  www.rand.org 

Explore  the  RAND  National  Defense  Research  Institute 

View  document  details 

Limited  Electronic  Distribution  Rights 

This  document  and  trademark(s)  contained  herein  are  protected  by  law  as  indicated  in  a  notice  appearing 
later  in  this  work.  This  electronic  representation  of  RAND  intellectual  property  is  provided  for  non¬ 
commercial  use  only.  Unauthorized  posting  of  RAND  electronic  documents  to  a  non-RAND  website  is 
prohibited.  RAND  electronic  documents  are  protected  under  copyright  law.  Permission  is  required  from 
RAND  to  reproduce,  or  reuse  in  another  form,  any  of  our  research  documents  for  commercial  use.  For 
information  on  reprint  and  linking  permissions,  please  see  RAND  Permissions. 

Report  Documentation  Page 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 

1.  REPORT  DATE 

2Q 2.  REPORT  TYPE 

3.  DATES  COVERED 

00-00-2014  to  00-00-2014 

4.  TITLE  AND  SUBTITLE 

The  RAND  Online  Measure  Repository  for  Evaluating  Psychological 
Health  and  Traumatic  Brain  Injury  Programs.  The  RAND  Toolkit, 

Volume  2 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROTECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

RAND  Corporation, National  Defense  Research  Institute, 1776  Main 

Street,  PO  Box  2138, Santa  Monica, CA, 90407-2138 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSOR/MONITOR'S  ACRONYM(S) 

11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

15.  SUBIECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF:  17.  LIMITATION  OF 

_ _ _  ABSTRACT 

18.  NUMBER  19a.  NAME  OF 

OF  PAGES  RESPONSIBLE  PERSON 

a.  REPORT  b.  ABSTRACT  c.  THIS  PAGE  Same  OS 

unclassified  unclassified  unclassified  Report  (SAR) 

81 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


This  report  is  part  of  the  RAND  Corporation  research  report  series.  RAND  reports 
present  research  findings  and  objective  analysis  that  address  the  challenges  facing  the 
public  and  private  sectors.  All  RAND  reports  undergo  rigorous  peer  review  to  ensure 
high  standards  for  research  quality  and  objectivity. 


The  RAND  Online 
Measure  Repository  for 
Evaluating  Psychological 
Health  and  Traumatic 
Brain  Injury  Programs 

The  RAND  Toolkit,  Volume  2 


Joie  D.  Acosta,  Kerry  A.  Reynolds,  Emily  M.  Gillen,  Kevin  Carter  Feeney, 
Carrie  M.  Farmer,  Robin  M.  Weinick 


RAND 


NATIONAL  DEFENSE  RESEARCH  INSTITUTE 


The  RAND  Online 
Measure  Repository  for 
Evaluating  Psychological 
Health  and  Traumatic 
Brain  Injury  Programs 

The  RAND  Toolkit,  Volume  2 


Joie  D.  Acosta,  Kerry  A.  Reynolds,  Emily  M.  Gillen,  Kevin  Carter  Feeney, 
Carrie  M.  Farmer,  Robin  M.  Weinick 


Prepared  for  the  Office  of  the  Secretary  of  Defense  and  the  Defense  Centers  of  Excellence 

for  Psychological  Health  and  Traumatic  Brain  Injury 

Approved  for  public  release;  distribution  unlimited 


This  research  was  sponsored  by  the  the  Office  of  the  Secretary  of  Defense  (OSD)  and  the 
Defense  Centers  of  Excellence  for  Psychological  Health  and  Traumatic  Brain  Injury.  It  was 
conducted  in  the  Forces  and  Resources  Policy  Center,  a  RAND  National  Defense  Research 
Institute  (NDRI)  program.  NDRI  is  a  federally  funded  research  and  development  center 
sponsored  by  the  OSD,  the  Joint  Staff,  the  Unified  Combatant  Commands,  the  Navy, 
the  Marine  Corps,  the  defense  agencies,  and  the  defense  Intelligence  Community  under 
Contract  W74V8H-06-C-0002. 


Library  of  Congress  Cataloging-in-Publication  Data  is  available  for  this  publication. 

ISBN:  978-0-8330-5938-3 


The  RAND  Corporation  is  a  nonprofit  institution  that  helps  improve  policy  and 
decisionmaking  through  research  and  analysis.  RAND’s  publications  do  not  necessarily 
reflect  the  opinions  of  its  research  clients  and  sponsors. 

Support  RAND  — make  a  tax-deductible  charitable  contribution  at 
www.rand.org/ giving/ contribute.html 

RAND®  is  a  registered  trademark. 


©  Copyright  2014  RAND  Corporation 

This  document  and  trademark^)  contained  herein  are  protected  by  law.  This  representation 
of  RAND  intellectual  property  is  provided  for  noncommercial  use  only.  Unauthorized 
posting  of  RAND  documents  to  a  non-RAND  website  is  prohibited.  RAND  documents 
are  protected  under  copyright  law.  Permission  is  given  to  duplicate  this  document  for 
personal  use  only,  as  long  as  it  is  unaltered  and  complete.  Permission  is  required  from 
RAND  to  reproduce,  or  reuse  in  another  form,  any  of  our  research  documents  for 
commercial  use.  For  information  on  reprint  and  linking  permissions,  please  see  the  RAND 
permissions  page  (www.rand.org/pubs/permissions.html). 

RAND  OFFICES 

SANTA  MONICA,  CA  •  WASHINGTON,  DC 
PITTSBURGH,  PA  •  NEW  ORLEANS,  LA  •  JACKSON,  MS  •  BOSTON,  MA 
CAMBRIDGE,  UK  •  BRUSSELS,  BE 

www.rand.org 


Preface 


Throughout  the  past  decade,  U.S.  military  forces  have  been  engaged  in  extended  conflicts 
in  Iraq  and  Afghanistan.  While  most  military  personnel  cope  well  across  the  deployment 
cycle,  increases  in  stress  associated  with  rapid  operational  tempo  may  raise  the  risk  for  mental 
health  challenges.  During  this  time,  the  U.S.  Department  of  Defense  (DoD)  has  implemented 
numerous  programs  to  support  service  members  and  their  families  as  they  cope  with  these 
challenges.  These  programs  address  various  components  of  biological,  psychological,  social, 
spiritual,  and  holistic  influences  on  psychological  health  along  the  resilience,  prevention,  and 
treatment  continuum  and  focus  on  a  variety  of  clinical  and  nonclinical  concerns.  In  response 
to  this  proliferation  of  programs,  the  Assistant  Secretary  of  Defense  for  Health  Affairs  asked 
the  RAND  National  Defense  Research  Institute  (NDRI)  to  develop  a  set  of  tools  to  support 
ongoing  assessment  and  evaluation  of  the  DoD  portfolio  of  programs  that  address  psychologi¬ 
cal  health  and  traumatic  brain  injury  (TBI). 

This  report  describes  the  development  and  uses  of  one  of  these  tools,  the  RAND  Online 
Measure  Repository  (ROMR).  The  ROMR  is  an  online  searchable  database  containing  mea¬ 
sures  related  to  psychological  health  and  TBI  that  was  created  to  support  monitoring  and 
evaluation  of  such  programs.  This  report  describes  the  rationale,  purpose,  and  uses  of  the 
ROMR,  as  well  as  the  content  of  the  repository  itself.  This  report  will  be  of  particular  interest 
to  DoD  officials  responsible  for  programs  related  to  psychological  health  and  TBI,  and  should 
also  be  helpful  for  health  policy  officials  within  the  U.S.  Department  of  Veterans  Affairs  (VA) 
as  well  as  others  responsible  for  evaluating  similar  programs  in  nondefense-related  settings.  The 
ROMR  is  the  second  part  of  a  toolkit  that  RAND  is  developing  to  support  the  evaluation  of 
this  DoD  portfolio  of  programs.  A  link  to  the  ROMR  is  available  at  the  “Innovative  Practices 
for  Psychological  Health  and  Traumatic  Brain  Injury”  web  page;1  other  tools  in  this  series  will 
be  made  available  at  this  site  as  they  are  prepared. 

This  research  was  sponsored  by  the  Office  of  the  Secretary  of  Defense  and  the  Defense 
Centers  of  Excellence  for  Psychological  Health  and  Traumatic  Brain  Injury  and  conducted 
within  the  Forces  and  Resources  Policy  Center  of  the  RAND  National  Defense  Research  Insti¬ 
tute,  a  federally  funded  research  and  development  center  sponsored  by  the  Office  of  the  Sec¬ 
retary  of  Defense,  the  Joint  Staff,  the  Unified  Combatant  Commands,  the  Navy,  the  Marine 
Corps,  the  defense  agencies,  and  the  defense  Intelligence  Community. 

For  more  information  on  the  RAND  Forces  and  Resources  Policy  Center,  see  http:// 
www.rand.org/nsrd/ndri/centers/frp.html  or  contact  the  director  (contact  information  is  pro¬ 
vided  on  the  web  page). 


http://www.rand.org/multi/military/innovative-practices.html 


l 


Contents 


Preface . iii 

Figure  and  Tables . vii 

Summary .  ix 

Acknowledgments . xiii 

Abbreviations . xv 

CHAPTER  ONE 

Introduction .  1 

The  Need  for  a  Repository  of  Measures  on  Psychological  Health  and  Traumatic  Brain  Injury  to 

Evaluate  Military  Programs . 2 

CHAPTER  TWO 

Development  of  the  RAND  Online  Measure  Repository . 5 

Identifying  Potential  Measures . 6 

Literature  Search . 7 

Expert  Consultation . 7 

Review  of  References . 7 

Selecting  Measures  from  the  Literature  Search  to  Include  in  the  Repository . 7 

Title  and  Abstract  Review . 7 

Full-Text  Review  for  Inclusion  and  Exclusion  Criteria . 8 

Articles  Identified  During  the  Literature  Search . 8 

Abstracting  Consistent  Information  on  Each  Measure . 9 

Article  Coding  and  Data  Abstraction . 9 

Procedure  for  Coding  Articles . 11 

CHAPTER  THREE 

Characteristics  of  Measures  Included  in  the  RAND  Online  Measure  Repository . 13 

Use  by  Military  Populations  and  Cost . 13 

Breadth  and  Length . 13 

Methods  of  Administration . 15 

Clinical  Significance . 15 

Psychometrics . 16 


v 


vi  The  RAND  Online  Measure  Repository  for  Evaluating  Psychological  Health  and  Traumatic  Brain  Injury  Programs 


CHAPTER  FOUR 

Potential  Uses  of  the  RAND  Online  Measure  Repository . 19 

Select  Measures  for  Program  Evaluation  or  Research  Related  to  Psychological  Health  and 

Traumatic  Brain  Injury . 19 

Select  Measures  for  Dual  Use  by  Both  Clinicians  and  Program  Evaluators  or  Researchers . 19 

Identify  Core  Outcome  Measures  for  Evaluating  Similar  Programs .  20 

Determine  Need  for  Additional  Development  or  Validation  of  Measures  with 

Military  Populations .  20 

Conclusions .  20 

APPENDIXES 

A.  Existing  Measure  Repositories . 21 

B.  Detailed  Literature  Search  Strategies  Used  to  Identify  Measures . 25 

C.  References  Included  in  the  RAND  Online  Measure  Repository . 29 

D.  List  of  Measures  Included  in  the  RAND  Online  Measure  Repository . 45 

E.  Data  Abstraction  Lorm . 51 

L.  Glossary . 55 

G.  Brief  Guide  to  Interpreting  Measure  Reliability  and  Validity  in  the  RAND  Online 

Measure  Repository . 57 

H.  Brief  User  Guide  for  the  RAND  Online  Measure  Repository . 59 

References . 61 


Figure  and  Tables 


Figure 

2.1.  Flowchart  for  Literature  Search . 10 

Tables 

2.1.  Key  Features  of  ROMR  Compared  to  Other  Repositories . 6 

2.2.  Keyword  Categories:  An  Example  Search  Strategy  for  TBI  Articles  in  PubMed . 8 

2.3.  Inclusion  and  Exclusion  Criteria . 9 

3.1.  Measures  Identified  for  Military  Populations  and  Available  at  No  Cost,  by  Domain .  14 

3.2.  Measures  Covering  Multiple  Domains .  14 

3.3.  Methods  Used  to  Administer  Measures .  15 

3.4.  Measures  with  Clinical  Significance  Scores,  by  Domain .  16 

3.5.  Measures  with  Published  Psychometrics,  by  Domain .  17 

B.l.  Detailed  Literature  Search  Strategies  Used  to  Identify  Measures .  25 

E.  1 .  Data  Abstraction  Form .  51 


vii 


Summary 


More  than  2.2  million  service  members  have  deployed  to  support  military  operations  in  Iraq 
and  Afghanistan  during  the  past  decade  (Levin,  2011).  Among  service  members  who  had  been 
deployed  to  Iraq  and  Afghanistan  as  of  October  2007,  approximately  one-hfth  reported  cur¬ 
rent  symptoms  consistent  with  posttraumatic  stress  disorder  (PTSD)  or  major  depression,  and 
about  the  same  number  reported  having  experienced  a  probable  traumatic  brain  injury  (TBI) 
while  deployed  (Tanielian  and  Jaycox,  2008).  DoD  has  implemented  numerous  programs, 
interventions,  and  policies  to  address  the  increased  concerns  about  PTSD  and  TBI  and  their 
effects  on  service  members  and  their  families.  These  programs  focus  on  reducing  the  incidence 
of  mental  health  problems  via  efforts  to  improve  readiness  and  resilience;  providing  infor¬ 
mation,  connecting  individuals  to  care,  and  encouraging  help  seeking;  identifying  individu¬ 
als  with  mental  health  concerns  or  TBI;  providing  or  improving  clinical  services,  or  offering 
mental  health  services  in  nontraditional  locations  to  expand  access  to  care;  providing  a  wide 
range  of  training  and  educational  activities;  and  supporting  service  members  and  their  families 
during  times  of  military  transition  (Weinick  et  ah,  2011). 

As  these  efforts  have  proliferated,  it  has  become  increasingly  important  to  evaluate  their 
effectiveness.  To  support  the  design  and  implementation  of  such  program  evaluations,  we  devel¬ 
oped  an  online  repository  of  measures  (the  RAND  Online  Measures  Repository,  or  ROMR) 
that  indexes  and  describes  measures  related  to  psychological  health  and  TBI  that  have  been 
used  in  both  civilian  and  military  populations.  Specifically,  we  identified  measures  of  primary 
importance  to  TBI  including  measures  of  cognition,  executive  functioning,  and  memory.  We 
have  also  identified  measures  of  primary  importance  to  psychological  health  in  the  following 
domains:  PTSD,  depression,  anxiety,  suicidal  ideation,  and  resiliency.  We  also  identified  mea¬ 
sures  relevant  to  military  units  such  as  unit  cohesion  and  force  readiness  and  preservation. 

The  ROMR  is  the  second  part  of  a  toolkit  that  RAND  is  developing  to  support  the  assess¬ 
ment  and  evaluation  of  the  DoD  portfolio  of  programs.  A  link  to  the  ROMR  is  available  at  the 
“Innovative  Practices  for  Psychological  Health  and  Traumatic  Brain  Injury”  web  page;1  other 
tools  in  this  series  will  be  made  available  at  this  site  as  they  are  prepared. 


Focus  of  This  Report 

This  report  describes  the  ROMR,  and  explains  how  it  was  developed  and  how  it  can  be  used. 
Chapter  One  introduces  the  report  by  describing  the  need  for  an  online  searchable  database 
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of  measures  to  support  monitoring  and  evaluation  of  psychological  health  and  TBI  programs. 
Chapter  Two  describes  the  development  of  the  ROMR,  including  the  rationale  for  its  creation 
and  the  method  used  to  identify  measures  and  extract  relevant  information.  Chapter  Three 
provides  a  description  of  the  measures  included  in  the  ROMR.  Chapter  Four  describes  the 
potential  benefits  of  the  ROMR  to  agency  officials,  program  managers,  mental  health  profes¬ 
sionals,  and  those  interested  in  program  evaluation. 


How  the  RAND  Online  Measure  Repository  Was  Developed 

The  ROMR  was  developed  using  a  series  of  literature  searches,  journal  reviews,  and  expert  rec¬ 
ommendations  to  identify  measures  of  anxiety,  depression,  PTSD,  resiliency,  suicidal  thoughts, 
unit  cohesion,  force  readiness  and  preservation,  and  measures  related  to  TBI.  Relevant  arti¬ 
cles  were  coded  using  a  standardized  abstraction  procedure  guided  by  supporting  documents 
(e.g.,  glossary  of  terms)  and  procedures  (weekly  discussion  of  coding  issues).  We  focused  on 
sources  that  described  the  development,  validation,  and/or  psychometric  properties  of  one  or 
more  measures.  From  each  source,  we  abstracted  information  about  the  measure’s  domains, 
administration,  scoring,  length,  acquisition,  and  psychometric  properties,  as  well  as  identified 
the  populations  to  which  the  measure  had  been  applied.  Once  information  on  measures  was 
abstracted  and  reviewed,  the  database  used  for  coding  was  converted  into  a  searchable  online 
tool. 


Measures  Included  in  the  RAND  Online  Measure  Repository 

We  identified  174  measures  including  a  wide  array  of  measures  of  depression  (71),  PTSD  (49), 
and  anxiety  (41).  Several  measures  related  to  exposure  to  traumatic  events  (21),  stress  and 
coping  (16),  resiliency  (15),  suicidal  thoughts  (16),  and  TBI  (e.g.,  cognition  functioning  and 
speech)  (20)  were  also  identified.  Fewer  measures  of  force  readiness  (4)  and  unit  cohesion  (10) 
were  identified,  suggesting  that  this  may  be  a  less  developed  held  of  measurement.  Eighty-four 
percent  of  the  measures  identified  had  been  used  with  adults,  and  23  percent  had  been  used 
with  children.  The  majority  of  measures  were  self- administered  questionnaires.  It  is  important 
to  note  that  the  purpose  of  the  ROMR  is  to  support  program  evaluation,  rather  than  clinical 
care,  so  we  have  not  included  measures  related  to  diagnosis  of  mental  health  disorders  or  TBI. 

The  measures  most  commonly  used  with  military  populations  were  those  related  to 
depression  and  PTSD.  Flowever,  only  about  half  of  the  total  measures  identified  had  ever  been 
used  with  a  military  population.  Additional  work  is  needed  to  validate  many  of  these  mea¬ 
sures  in  military  populations,  especially  measures  with  clinical  significance,  no-cost  measures 
of  anxiety,  and  measures  for  evaluation  of  programs  related  to  TBI.  As  updates  are  made  to 
the  ROMR,  additions  may  also  be  considered  to  continue  building  areas  of  the  ROMR  where 
fewer  measures  were  identified.  These  areas  include  measures  of  leadership,  force  readiness, 
unit  cohesion,  and  family  support. 


Summary  xi 


Potential  Uses  of  the  RAND  Online  Measure  Repository 

The  ROMR  has  a  number  of  potential  uses  across  a  wide  variety  of  programs  and  professionals. 

Select  Measures  for  Program  Evaluation  or  Research  Related  to  Psychological  Health  and 
Traumatic  Brain  Injury 

The  primary  purpose  of  the  ROMR  is  to  help  program  evaluators  select  appropriate  measures 
for  use.  Program  evaluators,  researchers,  and  those  responsible  for  program  implementation 
can  use  the  repository  to  identify  specific  measures  across  a  wide  variety  of  domains  related 
to  TBI  including  cognition,  executive  functioning,  and  memory,  and  psychological  health 
including  depression,  anxiety,  PTSD,  stress  and  coping,  and  resiliency,  among  others. 

Select  Measures  for  Dual  Use  by  Both  Clinicians  and  Program  Evaluators  or  Researchers 

The  ROMR  includes  information  on  available  clinical  cutoff  scores  used  to  determine  when 
individuals  require  clinical  services,  to  inform  clinical  case  planning,  or  to  screen  individuals 
who  may  be  at  risk  for  developing  a  psychological  disorder.  Measures  with  clinical  meaning 
may  be  useful  to  both  clinicians  providing  novel  interventions  or  other  services  specifically  tar¬ 
geting  clinical  outcomes  and  to  the  evaluators  or  researchers  working  with  these  clinicians  to 
determine  the  effectiveness  of  their  services. 

Identify  Core  Outcome  Measures  for  Evaluating  Similar  Programs 

Organizations  or  individuals  responsible  for  a  group  of  programs  could  consider  using  the 
ROMR  to  identify  and  endorse  a  specific  set  of  outcome  measures  that  are  both  reliable  and 
valid  for  the  populations  served  across  a  variety  of  domains.  Endorsing  a  specific  set  of  outcome 
measures  could  allow  for  consistency  in  tracking  core  outcomes  or  indicators  of  effectiveness 
across  an  array  of  programs. 

Determine  Need  for  Additional  Reliability  and  Validity  Testing  of  Measures  with  Military 
Populations 

The  ROMR’s  assessment  of  measures  currently  used  by  program  evaluators  and  researchers 
can  be  helpful  in  determining  where  more  work  needs  to  be  done  to  establish  reliability  and 
validity  of  measures  with  military  populations.  Additional  psychometric  development  is  par¬ 
ticularly  important  for  domains  such  as  force  readiness,  where  only  a  few  measures  exist,  and 
domains  such  as  anxiety,  where  there  has  been  little  testing  with  military  populations. 

Conclusion 

Valid  and  reliable  measures  of  psychological  health  and  TBI-related  constructs  are  needed  to 
be  able  to  monitor  and  evaluate  programs  that  address  these  issues.  The  ROMR  is  a  valuable 
tool  that  responds  to  this  need  by  providing  an  online  and  searchable  database  of  measures 
related  to  psychological  health  and  TBI. 
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CHAPTER  ONE 


Introduction 


Our  nation’s  all-volunteer  military  force  continues  to  endure  the  longest  era  of  conflict  in  its 
history.  The  past  decade  has  been  characterized  by  frequent  deployments  and  exposure  to 
combat-related  trauma,  which  have  increased  the  risk  of  postdeployment  psychological  health 
problems  among  the  more  than  2.2  million  troops  that  have  deployed  since  2001  (Levin, 
2011).  Concerns  have  been  raised  about  the  incidence  of  posttraumatic  stress  disorder  (PTSD), 
major  depression,  traumatic  brain  injury  (TBI),  and  suicide  among  returning  service  members. 
Research  has  shown  that  between  15  and  20  percent  of  returning  service  members  reported 
symptoms  consistent  with  current  PTSD  or  major  depression.  Similarly,  about  20  percent 
reported  having  experienced  a  probable  TBI  while  deployed  (Tanielian  and  Jaycox,  2008). 
These  medical  and  psychological  health  issues  affect  both  service  members  and  their  families 
(Tanielian  and  Jaycox,  2008;  Chandra  et  ah,  2010). 

In  response  to  growing  concerns  about  the  effects  of  deployment  on  service  members, 
the  Department  of  Defense  (DoD)  Task  Force  on  Mental  Health  recommended  that  DoD 
implement  an  array  of  programs  to  address  emerging  psychological  health  issues  (Department 
of  Defense  Task  Force  on  Mental  Health,  2007).  In  response  to  these  recommendations,  DoD 
has  implemented  programs,  interventions,  and  policies  to  improve  readiness  and  resilience,  to 
improve  awareness  and  understanding  of  psychological  health  issues  and  TBI,  and  to  increase 
access  to  care  and  enhance  existing  services.  Programs  and  interventions  focus  on: 

•  identifying  individuals  with  psychological  health  issues  or  TBI 

•  encouraging  help-seeking  behavior 

•  providing  support  to  service  members  during  times  of  military  transition 

•  providing  or  enhancing  existing  clinical  services  programs 

•  providing  care  in  nontraditional  locations 

•  providing  training  and  education  activities  to  improve  the  capacity  of  mental  health  pro¬ 
viders.  (Weinick  et  ah,  2011) 

To  help  DoD  appropriately  monitor  the  effectiveness  of  these  programs,  we  developed 
the  RAND  Online  Measure  Repository  (ROMR)  to  increase  the  information  available  on 
measures  that  can  support  program  evaluation  in  these  areas.  The  ROMR  is  the  second  part 
of  a  toolkit  that  RAND  is  developing  to  support  the  evaluation  of  the  DoD  portfolio  of  pro¬ 
grams.  A  link  to  the  ROMR  is  available  at  the  “Innovative  Practices  for  Psychological  Health 
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and  Traumatic  Brain  Injury”  web  page;1  other  tools  in  this  series  will  be  made  available  at  this 
site  as  they  are  prepared. 


The  Need  for  a  Repository  of  Measures  on  Psychological  Health  and 
Traumatic  Brain  Injury  to  Evaluate  Military  Programs 

The  U.S.  Government  Accountability  Office  (GAO)  defines  program  evaluation  as  individual 
systematic  studies  conducted  periodically  or  on  an  ad  hoc  basis  to  “assess  whether  the  pro¬ 
gram  works  .  .  .  and  identify  adjustments  that  may  improve  its  results”  (GAO,  2005,  p.  3). 
The  purpose  of  program  evaluation  is  to  determine  if  a  program  is  a  worthwhile  investment 
and  is  achieving  its  intended  impact,  and  to  provide  accountability  to  program  funders  and 
the  public.  Ongoing  evaluation  can  serve  as  an  early  warning  system  for  program  funders  to 
identify  adjustments  to  program  approaches  that  may  improve  their  results. 

Although  program  evaluations  may  take  many  forms,  a  key  piece  of  any  evaluation  design 
is  the  careful  selection  of  measures  that  are  appropriate  for  the  population  served  by  the  pro¬ 
gram  (e.g.,  service  members  and  spouses  or  children  of  service  members)  and  for  the  outcomes 
being  targeted  for  improvement  (e.g.,  reducing  depression,  building  resiliency).  Measures  assess 
“the  type  or  level  of  program  activities  conducted  (process),  the  direct  products  and  services 
delivered  by  a  program  (outputs),  and/or  the  results  of  those  products  and  services  (outcomes)” 
and  can  be  in  the  format  of  a  questionnaire  or  inventory,  structured  or  unstructured  interview, 
or  observation  rating  tool  (GAO,  2005,  p.  4). 

One  key  challenge  in  conducting  rigorous  evaluations  is  identifying  and  selecting  appro¬ 
priate  evaluation  measures.  The  creation  of  a  single  database  of  potential  measures  with  infor¬ 
mation  about  their  use  can  support  program  evaluation  activities  by  reducing  the  burden  on 
program  staff  to  identify  and  select  appropriate  measures. 

Existing  measure  repositories  are  limited  because  they  lack  information  on  whether  mea¬ 
sures  have  been  used  with  a  military  population,  do  not  include  measures  related  to  TBI,  or 
require  a  fee  for  usage  (Chapter  Two  contains  a  review  of  existing  repositories).  To  address 
these  issues  and  support  the  design  and  implementation  of  evaluations  of  psychological  health 
and  TBI  programs,  we  developed  the  ROMR.  This  publicly  available  online  repository  indexes 
and  describes  measures  related  to  psychological  health  and  TBI  that  have  been  used  in  both 
civilian  and  military  populations,  as  well  as  measures  that  are  specifically  relevant  to  military 
units  (e.g.,  unit  cohesion,  force  readiness  and  preservation).  We  identified  measures  of  primary 
importance  to  TBI  including  measures  of  cognition,  executive  functioning,  and  memory.  We 
have  also  included  measures  of  primary  importance  to  psychological  health  in  the  domains 
of  PTSD,  depression,  anxiety,  military  unit  measures,  suicidal  ideation,  and  resiliency.  It  is 
important  to  note  that  the  purpose  of  the  ROMR  is  to  support  program  evaluation,  rather 
than  clinical  care,  so  we  have  not  included  measures  related  to  diagnosis  of  TBI. 

The  ROMR  contains  174  measures  related  to  psychological  health  and  TBI,  including 
descriptions  of  each  measure,  information  about  their  development  and  uses  (e.g.,  whether 
they  have  been  used  with  a  military  population),  as  well  as  documentation  on  their  reliability 
and  validity  and  other  characteristics  (e.g.,  length  of  measure,  cost).  Measure  descriptions  are 
indexed  using  an  online  interface  (accessible  at  the  “Innovative  Practices”  web  page)  that  allows 
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repository  users  to  search  for  specific  measures  based  on  content-relevant  keywords  or  other 
characteristics  of  each  measure  (e.g.,  whether  the  measure  has  published  psychometrics)  and  to 
compare  measures  side  by  side.  Users  can  search  by  keyword  or  other  features  and  can  conduct 
side-by-side  comparisons  of  measures  to  identify  the  best  options  for  use  in  program  evalua¬ 
tion.  The  ROMR  provides  a  targeted,  user-friendly  tool  to  support  the  design  and  implementa¬ 
tion  of  evaluations  of  new  or  existing  programs  related  to  psychological  health  and  TBI.  The 
online  interface  is  intended  as  the  most  useful  way  to  access  the  cataloged  information  about 
each  measure. 

This  report  provides  documentation  on  the  ROMR  to  help  potential  users  understand 
its  creation  and  to  explain  how  it  can  be  used.  Chapter  2  describes  the  development  of  the 
ROMR,  including  the  method  we  followed  in  identifying  measures  and  extracting  relevant 
information.  Chapter  3  provides  a  description  of  the  measures  included  in  the  ROMR.  Chap¬ 
ter  4  describes  the  potential  benefits  of  the  ROMR  to  agency  officials,  program  managers, 
mental  health  professionals,  and  those  interested  in  program  evaluation. 

A  series  of  appendixes  specifies  the  contents  of  the  ROMR,  provides  additional  detail  on 
the  development  of  ROMR,  and  includes  supporting  materials  to  aid  potential  users.  Appen¬ 
dix  A  contains  a  brief  description  of  existing  measure  repositories  that  were  reviewed  to  inform 
the  development  of  the  ROMR.  Appendix  B  describes  the  detailed  literature  search  strategies 
used  to  identify  relevant  measures.  Appendix  C  displays  a  comprehensive  list  of  the  refer¬ 
ences  consulted  in  developing  the  ROMR,  and  Appendix  D  lists  the  measures  included  in 
the  ROMR.  Appendix  E  describes  in  detail  how  information  on  each  measure  was  abstracted 
by  RAND  staff,  and  Appendix  F  contains  a  glossary  of  relevant  terms.  Appendix  G  pro¬ 
vides  information  to  help  users  interpret  reliability  and  validity  information  contained  in  the 
ROMR.  Finally,  a  brief  guide  for  how  to  access  and  search  the  ROMR  comprises  Appendix  H. 
Complete  information  on  each  measure  included  in  the  ROMR  is  available  at  the  “Innovative 
Practices”  web  page. 


CHAPTER  TWO 


Development  of  the  RAND  Online  Measure  Repository 


The  first  step  in  developing  the  ROMR  was  to  conduct  a  search  of  existing  measure  reposito¬ 
ries,  which  were  identified  through  a  web-based  search  as  well  as  personal  communication  with 
experts  in  the  fields  of  psychological  measurement  and  military  mental  health.  This  process 
identified  eight  repositories  that  contained  measures  related  to  TBI,  psychological  health  at  the 
individual  or  unit  level,  or  resiliency:  the  ADAI  Substance  Use  Assessment  Instrument  Library; 
the  Buros  Institute  of  Mental  Measurements;  the  Center  for  Outcome  Measurement  in  Brain 
Injury;  the  Compendium  of  Assessment  and  Research  Tools;  the  ETS  Test  Collection/Carl  C. 
Brigham  Library;  Measuring  Violence-Related  Attitudes,  Behaviors,  and  Influences  Among 
Youths:  A  Compendium  of  Assessment  Tools,  2nd  ed.;  the  National  Center  for  the  Study  of 
PTSD;  and  the  Patient  Reported  Outcome  and  Quality  of  Life  Instruments  Database. 

We  reviewed  the  eight  existing  repositories  to  determine  the  type  of  measures  they 
included  and  examined  several  features,  including  access  fees,  inclusion  of  measures  used  with 
military  populations,  information  about  obtaining  measures,  and  psychometric  properties 
described  by  the  repositories.  We  found  that  each  had  a  number  of  challenges  that  would 
limit  its  potential  direct  use  for  evaluating  military  programs.  For  example,  some  repositories 
included  information  on  how  to  obtain  the  measure  or  the  psychometric  qualities  of  the  mea¬ 
sure,  but  did  not  contain  information  on  whether  the  measure  had  been  used  with  a  military 
population.  We  could  not  locate  a  repository  that  included  a  combination  of  measures  related 
to  TBI,  psychological  health  at  both  the  individual  and  unit  levels,  and  resiliency,  and  we 
found  only  one  repository  that  contained  information  about  whether  the  measures  had  been 
used  with  a  military  population.  Furthermore,  many  military  programs  focus  on  improving 
the  psychological  health  of  military  family  members,  but  it  was  difficult  to  find  measures  for 
both  adults  and  children  within  many  of  the  existing  repositories.  Appendix  A  describes  the 
existing  measure  repositories  in  greater  detail. 

Developed  to  address  the  limitations  of  existing  repositories,  the  ROMR  differs  from 
other  available  databases  in  the  content  area  covered,  populations  targeted,  and  the  audience 
for  which  it  was  designed.  The  ROMR  focuses  on  measures  related  to  TBI  for  adults  and  mea¬ 
sures  related  to  psychological  health  for  adults,  adolescents,  and  children  in  both  civilian  and 
military  populations.  In  addition,  while  existing  databases  may  provide  some  content  free  of 
charge,  the  entire  content  of  the  ROMR  is  publicly  available  at  no  cost.  Finally,  the  ROMR 
is  designed  for  the  purpose  of  program  evaluation,  with  measures  focused  on  outcomes  rel¬ 
evant  to  TBI  including  cognition,  executive  functioning,  and  memory,  and  measures  focused 
on  mental  health,  including  depression,  PTSD,  and  resiliency.  These  features  are  designed  to 
improve  user-friendliness  and  efficiency  by  reducing  the  examination  of  less  relevant  materials 
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and  increasing  quick  identification  of  the  most  relevant  measures.  Table  2.1  compares  ROMR 
features  to  other  existing  measure  repositories. 

Once  we  determined  the  required  features  of  the  ROMR,  we  began  a  systematic  process 
to  identify  measures  related  to  TBI,  psychological  health,  unit  cohesion,  and  resiliency  needed 
to  populate  the  repository. 


Identifying  Potential  Measures 

We  searched  the  peer-reviewed  literature  to  identify  measures  of  anxiety,  depression,  PTSD, 
TBI  (e.g.,  cognition,  executive  functioning,  memory),  resiliency,  suicidal  thoughts,  unit 


Table  2.1 

Key  Features  of  ROMR  Compared  to  Other  Repositories 
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cohesion,1  and  force  health  readiness.2  To  identify  relevant  articles,  we  began  with  keyword 
searches  of  content-relevant  databases,  followed  by  title  and  abstract  review.  Additional  mea¬ 
sures  were  identified  during  article  abstraction  and  by  content-area  experts.  We  focused  on 
sources  that  described  the  development,  validation,  and/or  psychometric  properties  of  one  or 
more  measures. 

Literature  Search 

We  conducted  keyword  searches  in  three  databases  that  focus  on  substantive  areas  pertaining 
to  mental  health  and  psychology,  medicine,  and  traumatic  stress:  PsychINFO  (psychology), 
PubMed  (medicine),  and  the  Published  International  Literature  on  Traumatic  Stress  database 
(PILOTS,  traumatic  stress).  We  also  searched  for  articles  published  in  the  last  two  years  in  the 
journal  Military  Psychology. 

Multiple  searches  were  conducted  in  each  database  to  ensure  that  references  from  all 
relevant  substantive  domains  were  included.  We  restricted  our  search  to  articles  published  in 
English  during  or  after  2000,  and  excluded  editorials,  letters,  and  commentaries.  Search  strat¬ 
egies  varied  for  each  substantive  domain  and  were  based  on  the  constraints  of  each  electronic 
database.  However,  most  searches  were  variations  on  a  basic  three-category  format:  The  first 
category  focused  on  keywords  related  to  the  substantive  domain,  the  second  category  focused 
on  keywords  relevant  for  establishing  psychometric  properties,  and  the  third  category  focused 
on  keywords  related  to  the  measurement  techniques  relevant  to  that  domain.  Details  of  the 
search  strategies  can  be  viewed  in  Appendix  B,  and  an  example  follows  in  Table  2.2. 

Expert  Consultation 

The  content  area  for  some  of  the  measures  included  in  the  repository  represent  emerging  areas 
of  research — specifically,  unit  cohesion  and  force  health  readiness.  For  these  areas,  content  area 
experts  were  asked  to  identify  additional  relevant  articles  or  measures  that  may  not  have  been 
identified  in  initial  literature  searches. 

Review  of  References 

Additional  articles  and  measures  were  identified  by  reviewing  the  references  in  articles  identi¬ 
fied  for  inclusion  in  the  repository. 


Selecting  Measures  from  the  Literature  Search  to  Include  in  the  Repository 

Title  and  Abstract  Review 

From  the  list  of  articles  identified  during  our  literature  search,  we  reviewed  titles  to  remove 
those  that  were  clearly  irrelevant  to  the  current  project.  For  the  remaining  articles,  we  reviewed 


1  Unit  cohesion  is  defined  as  the  bonding  together  of  members  of  an  organization  in  such  a  way  as  “to  sustain  their  will 
and  commitment  to  each  other,  their  unit  and  the  mission”  (Powell  et  al.,  2006). 

2  Force  health  readiness  is  defined  as  the  ability  of  U.S.  military  forces  to  “optimize  and  protect  the  psychological  and 
physical  health  of  service  members  and  their  families  through  policies  and  programs  across  all  phases  of  deployment.”  Force 
health  protection  and  readiness  is  also  referred  to  as  force  protection,  force  preservation,  or  force  preservation  and  readiness 
(Office  of  the  Deputy  Assistant  Secretary  of  Defense,  2011). 


8  The  RAND  Online  Measure  Repository  for  Evaluating  Psychological  Health  and  Traumatic  Brain  Injury  Programs 


Table  2.2 

Keyword  Categories:  Example  Search  Strategy  for  TBI  Articles  in  PubMed 


Category  1: 

Substantive  Domain 

Category  2: 

Psychometric  Properties 

Category  3: 

Measurement  Techniques 

•  traumatic  brain  injury 

•  validation  studies  as  topic 

• 

survey 

•  psychometrics 

[MeSH] 

• 

surveys 

•  aptitude  tests 

•  valid* 

• 

instrument 

•  digit  span 

•  reliab* 

• 

instruments 

•  language  tests 

• 

questionnaire 

•  hearing  tests 

• 

questionnaires 

•  neuropsychological 

• 

psychiatric  status 

tests 

rating  scales 

•  retention  [MeSH] 

• 

measure 

•  psychomotor 

• 

measures 

performance 

•  sociometric  techniques 

NOTE:  The  asterisk  (*)  denotes  a  wildcard  search.  MeSH  is  an  abbreviation  for  medical  subject  heading. 


abstracts  and  categorized  articles  into  three  groups:  articles  that  were  clearly  irrelevant  (e.g., 
addressed  a  topic  area  not  covered  in  the  ROMR),  articles  that  should  move  forward  to  full- 
text  review,  and  those  that  were  questionable.  Questionable  articles  were  discussed  among  the 
team  and  a  decision  was  made  to  include  or  exclude  them.  An  electronic  record  of  inclusion 
and  exclusion  criteria  was  updated  as  decisions  were  made. 

Full-Text  Review  for  Inclusion  and  Exclusion  Criteria 

Articles  identified  for  full-text  review  were  carefully  examined  for  information  relevant  to 
inclusion  and  exclusion  criteria.  Articles  that  met  inclusion  criteria  were  then  coded  according 
to  the  process  outlined  in  Table  2.3. 

Articles  Identified  During  the  Literature  Search 

The  database  searches  and  review  of  Military  Psychology  yielded  2,083  unique  sources.  Title 
review  excluded  1,350  articles.  Of  733  articles  that  underwent  abstract  review,  471  were 
excluded.  Another  80  articles  were  excluded  during  full-text  review.  The  title  and  full-text 
reviews  to  determine  inclusion/exclusion  were  conducted  by  a  librarian,  two  research  assistants, 
and  two  RAND  researchers  with  expertise  in  psychology,  economics,  and  health  policy.  The 
primary  reason  for  exclusion  during  full-text  review  was  that  the  article  addressed  a  foreign 
population,  which  typically  was  not  revealed  in  the  title  or  abstract.  All  remaining  articles 
(n  =  290)  were  included  in  the  review.  A  flowchart  depicting  the  identification  and  exclusion 
of  references  is  shown  in  Figure  2.1. 

Although  290  articles  were  ultimately  included,  multiple  articles  often  addressed  the  psy¬ 
chometric  properties  of  a  single  measure.  Furthermore,  some  articles  contained  information 
about  multiple  measures.  The  final  ROMR  contains  174  measures.  Appendix  C  contains  a  list 
of  the  articles  reviewed,  and  Appendix  D  contains  a  list  of  the  unique  measures  included  in 
the  database. 
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Table  2.3 

Inclusion  and  Exclusion  Criteria 


Inclusion  Criteria 


•  Contained  a  reference  to  at  least  one  of  the  following  eight  domains: 

1.  Depression,  including  dysthymia,  anhedonia,  bipolar  disorder,  cyclothymia,  and  bereavement  but  excluding 
postpartum  depression 

2.  Anxiety,  including  general  anxiety,  social  anxiety,  obsessive  compulsive  disorder,  panic  disorder,  phobias, 
and  worry  but  excluding  specific  anxieties  such  as  fear  of  pain,  fear  of  death,  fear  of  hospitalization,  etc. 

3.  PTSD,  including  measures  of  exposure  to  trauma,  internalizing  and  externalizing  behaviors  associated 
with  trauma,  symptoms  of  PTSD,  acute  stress  reaction,  acute  stress  disorder,  combat  and  operational  stress 
reaction,  reexperiencing  traumatic  events,  avoidance  of  place/things  reminiscent  of  trauma,  emotional 
numbing,  and  increased  arousal 

4.  TBI,  including  diagnosis  of  insults,  head  injuries,  head  trauma,  closed  head  wounds,  concussions,  and  other 
acquired  brain  trauma,  cognitive  and  motor  abilities 

5.  Suicidal  thoughts,  including  ideations,  intent,  and  fantasies 

6.  Resiliency,  including  coping  with  stress  and  hardiness 

7.  Force  health  readiness 

8.  Unit  cohesion,  including  unit  support. 

•  Addressed  some  aspect  of  psychometric  evaluation  of  the  focus  measure  (e.g.,  calculation  of  scale  reliability, 
test  of  validity) 

•  Focused  on  one  or  more  of  the  following  three  subpopulations:  (1)  U.S.  and  limited  international  military 
populations;  (2)  major  U.S.  racial/ethnic  groups  (e.g.,  white,  African  American,  Latino  American,  etc.);  (3) 
general  inpatient  populations  at  hospitals,  psychiatric  hospitals  and  rehabilitation  facilities.  Search  parameters 
were  extended  to  include  British,  Canadian,  Australian,  and  Israeli  military  populations  because  there  was 
limited  literature  on  U.S.  military  populations. 


Exclusion  Criteria 


•  Editorials,  letters,  and  commentaries 

•  Published  in  a  language  other  than  English 

•  Focused  solely  on  international  populations  with  the  exception  of  British,  Canadian,  Australian,  and  Israeli 
military  populations 

•  Focused  solely  on  individuals  over  age  50  or  retirees,  as  well  as  studies  where  the  sample  population  had  a 
mean  age  of  50  or  above.  The  majority  of  active  duty  service  members  are  under  50,  consequently  measures 
created  for  use  with  individuals  over  50  would  not  be  applicable  to  the  majority  of  active  duty  service 
members. 

•  Examined  a  specialized  subpopulation  (e.g.,  cancer  patients  with  depression,  patients  with  chronic  pain,  or 
specific  inpatient  populations) 

•  Addressed  TBI  solely  in  children  under  18  because  TBI  is  an  injury  primarily  acquired  by  service  members  in 
theater,  not  their  children. 


Abstracting  Consistent  Information  on  Each  Measure 

Article  Coding  and  Data  Abstraction 

The  first  two  authors  and  two  coders  abstracted  the  information  necessary  for  populating  the 
ROMR  from  each  included  article.  Each  piece  of  abstracted  information  represents  a  charac¬ 
teristic  or  quality  of  the  measure  that  potential  users  might  find  helpful  during  measure  selec¬ 
tion.  Abstracted  information  included  the  following: 

•  Substantive  domains:  as  many  domains  as  were  relevant  to  the  measure;  for  example,  a 
measure  assessing  symptoms  of  both  anxiety  and  depression 
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Figure  2.1 

Flow  Chart  for  Literature  Search 


NOTE:  More  than  one  measure  may  be  addressed  in  a 
single  article,  and  more  than  one  article  may  address  a 
specific  measure. 

RAND  RR487Z2-2.1 


•  Populations:  information  about  the  age  groups  (adult  or  child)  with  which  the  measure 
has  been  used  and  whether  or  not  the  measure  has  been  used  with  a  military  population 

•  Measure  administration:  including  the  method  of  administration  (e.g.,  questionnaire, 
interview),  the  person  who  administers  the  measure  (e.g.,  self,  trained  clinician),  and  the 
respondent  (e.g.,  self,  caregiver) 

•  Scoring:  information  about  subscales,  response  options,  and  the  presence  or  absence  of  a 
clinical  cutoff  score 

•  Measure  length:  number  of  items  in  the  measure 

•  Psychometric  properties:  including  reports  of  reliability  and  validity 

•  Measure  developer:  information  about  who  developed  the  measure  and  the  reference  for 
the  original  publication  of  the  measure  if  available 
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•  Measure  acquisition:  how  to  obtain  the  measure  and  if  there  is  a  fee  to  obtain  or  use  the 

measure. 

Further  detail  on  abstracted  information  is  available  in  the  data  abstraction  form  in 
Appendix  E. 

Procedure  for  Coding  Articles 

Coders  received  initial  instruction  on  the  use  of  the  data  abstraction  form  and  content  to  be 
included  for  each  domain  of  interest,  and  then  each  coded  five  articles,  which  were  reviewed 
by  this  report’s  first  two  authors.  This  process  was  repeated  until  the  coders  successfully  coded 
articles  independently  and  with  a  high  degree  of  reliability  among  team  members.  The  team 
regularly  reviewed  questions  about  coding;  Appendix  F  is  a  glossary  of  terms  developed  by  the 
team  to  support  the  coding  effort.  Once  all  articles  were  coded,  the  first  two  authors  reviewed 
all  measure  descriptions  for  completeness  of  information  and  clarity. 

During  this  review,  we  found  that  while  the  search  strategy  identified  a  wide  range  of 
measures  currently  in  use,  it  failed  to  yield  psychometric  information  for  55  of  the  measures 
and  did  not  adequately  distinguish  whether  each  of  these  measures  had  ever  been  used  by  a 
military  population  for  over  half  of  the  measures.  To  augment  these  two  domains,  authors 
engaged  the  existing  team  and  a  quantitative  psychologist  with  expertise  in  psychometrics  to 
conduct  a  second,  more  targeted  search  of  the  same  databases  used  in  the  initial  search  strategy, 
relaxing  the  inclusion  and  exclusion  criteria.  To  locate  additional  psychometric  information 
on  measures  for  which  information  was  missing,  a  search  was  conducted  using  the  name  of 
the  measure  and  the  keywords  reliability,  validity,  and  psychometrics.  To  augment  information 
on  whether  each  measure  had  ever  been  used  with  a  military  population,  the  team  conducted 
additional  literature  searches  using  keywords  including  the  name  of  the  measure  and  Army, 
Navy,  Marine  Corps,  Air  Force,  service  member,  and  military.  This  targeted  search  produced 
additional  information  that  allowed  us  to  determine,  for  all  measures,  whether  they  had  ever 
been  used  with  a  military  population.  We  collected  additional  information  about  use  with  a 
military  population  for  47  measures,  and  about  psychometric  evidence  for  an  additional  43 
measures.  The  final  set  of  all  entries  can  be  viewed  in  the  online  ROMR,  at  the  “Innovative 
Practices”  web  page. 


CHAPTER  THREE 


Characteristics  of  Measures  in  the  RAND  Online  Measure 
Repository 


We  identified  174  measures  including  a  wide  array  of  measures  of  depression  (71),  PTSD  (49), 
and  anxiety  (41).  Several  measures  related  to  exposure  to  traumatic  events  (21),  stress  and 
coping  (16),  resiliency  (15),  suicidal  thoughts  (16),  and  TBI  (20)  were  also  identified.  Fewer 
measures  of  force  readiness  (4)  and  unit  cohesion  (10)  were  identified,  suggesting  that  this  may 
be  a  less  developed  held  of  measurement.  Eighty-four  percent  of  the  measures  identified  had 
been  used  with  adults,  and  23  percent  had  been  used  with  children. 

In  this  chapter,  we  describe  the  characteristics  of  the  measures  included  in  the  ROMR 
based  on  their  use  in  military  populations,  availability  (cost),  breadth  (covering  a  single  domain 
or  multiple)  and  length,  mode  of  administration,  clinical  significance,  and  psychometrics. 
Appendix  D  contains  a  complete  list  of  the  measures  included  in  the  repository. 


Use  by  Military  Populations  and  Cost 

Two  features  of  the  repository  are  particularly  useful  for  evaluating  programs  for  military  pop¬ 
ulations.  First,  determining  whether  a  measure  has  been  used  in  military  populations  shows 
that  it  has  been  tested  already  as  directly  applicable  to  those  populations.  Second,  understand¬ 
ing  whether  there  is  a  cost  associated  with  using  a  particular  measure  can  help  researchers  and 
program  evaluators  plan  for  resources  that  may  be  needed  to  use  the  measure.  Programs  with 
more  limited  resources  may  wish  to  select  measures  that  are  available  at  no  cost. 

Table  3.1  summarizes  the  number  of  measures,  by  domain,  that  have  been  used  with 
military  populations,  and  the  number  available  at  no  cost.  As  previously  mentioned,  measures 
of  depression  (54),  PTSD  (33),  and  anxiety  (26)  were  among  the  most  common  measures  iden¬ 
tified  and  were  frequently  available  at  no  cost.  Similarly,  measures  of  depression  and  PTSD 
were  the  most  commonly  used  with  military  populations.  Although  only  10  measures  of  unit 
cohesion  were  identified,  as  expected,  all  had  been  used  with  military  populations. 


Breadth  and  Length 

When  selecting  measures  for  program  evaluation,  program  staff  must  often  balance  the  length 
of  time  required  or  the  space  on  a  questionnaire  needed  to  use  a  measurement  instrument 
against  the  ability  to  capture  multiple  outcomes  of  interest.  Some  efficiencies  may  be  gained  by 
using  a  single  measure  that  has  subscales  to  specifically  assess  well-being  in  multiple  domains. 
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Table  3.1 

Measures  Identified  for  Military  Populations  and  Available  at  No  Cost,  by  Domain 


Domain 

Number  of  Measures 
Identified 

Number  of  Measures 
Used  with  Military 
Populations 

Number  of  Measures 
Available  at  No  Cost 

PTSD 

49 

36 

33 

Depression 

71 

31 

54 

Anxiety 

41 

18 

26 

TBI 

20 

13 

8 

Suicidal  thoughts 

16 

7 

7 

Resiliency 

15 

10 

10 

Stress  and  coping 

16 

10 

10 

Exposure  to  traumatic  events 

21 

15 

15 

Force  readiness 

4 

4 

3 

Unit  cohesion 

10 

10 

7 

NOTE:  Measures  can  cover  more  than  one  domain;  therefore,  numbers  in  this  table  do  not  represent 
unique  measures  and  the  table  columns  cannot  be  summed.  Multiple  versions  of  a  single  measure  (e.g.,  a 
two-item  version  and  a  one-item  version)  were  counted  as  unique  measures. 


Of  the  measures  included  in  the  ROMR,  30  percent  address  two  or  more  domains  (e.g.,  depres¬ 
sion  and  anxiety),  with  the  remainder  of  measures  addressing  a  single  domain  (e.g.,  depres¬ 
sion).  Table  3.2  summarizes  both  the  total  number  of  measures  covering  one,  two,  three,  or 
four  or  more  domains,  and  the  mean  number  of  items  for  each  type  of  measure.  As  expected, 
measures  covering  a  single  domain  typically  consisted  of  fewer  items  than  measures  covering 
two  or  more  domains,  although  there  was  wide  variability. 


Table  3.2 

Measures  Covering  Multiple  Domains 


Number  of 

Domains  Covered 
by  Measure 

Number  of  Items 

Number  of  Measures 
Identified 

Mean 

Standard 

Deviation 

Range 

1 

122 

22.50 

26.5 

1-169 

2 

31 

38.10 

31.5 

2-120 

3 

14 

58.21 

66.4 

6-207 

4+ 

7 

220.70 

223.2 

5-567 
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Methods  of  Administration 

Ensuring  that  programs  have  appropriate  staff  to  administer  measures  is  another  key  factor 
to  consider  when  selecting  evaluation  measures.  Some  measures  can  only  be  administered 
through  an  interview  or  via  observation  by  a  clinician  or  trained  professional.  For  that  reason, 
many  program  evaluators  select  measures  that  are  self-administered  through  pen-and-paper 
or  online  assessments.  Table  3.3  summarizes  both  the  measure  administrator  and  the  method 
of  administration  for  the  measures  included  in  the  ROMR.  The  majority  of  measures  iden¬ 
tified  were  self-administered  questionnaires.  Over  half  of  these  had  been  used  with  military 
populations. 


Clinical  Significance 

Programs  providing  novel  interventions  or  other  services  specifically  targeting  clinical  out¬ 
comes  may  need  measures  that  provide  information  useful  for  both  program  evaluation  and 
clinical  case  planning  or  for  screening  of  at-risk  individuals.  About  one-fifth  of  the  measures 
in  the  ROMR  have  clinical  cutoff  scores  used  to  assign  clinical  meaning  to  an  individual’s 
responses.  Table  3.4  summarizes  the  domains  captured  by  measures  in  the  ROMR  with  clini- 


Table  3.3 

Methods  Used  to  Administer  Measures 


Administration  of  Measures 

Number  of  Measures 
Identified 

Number  of  Measures  Used 
with  Military  Populations 

Administrator 

Self-administered 

127 

75 

Administered  by  clinician 

25 

13 

Administered  by  trained  professional 
or  staff  member 

30 

17 

Other 

12 

6 

Method 

Survey  questionnaire  (pen  and  paper) 

125 

76 

Survey  questionnaire  (online) 

14 

8 

Structured  interview 

16 

6 

Semi-structured  interview 

18 

7 

Other 

22 

13 

NOTE:  A  single  measure  could  be  available  in  both  pen-and-paper  and  online  questionnaire 
formats;  therefore,  the  numbers  in  this  table  do  not  represent  unique  measures.  Multiple 
versions  of  a  single  measure  (e.g.,  a  two-item  version  and  a  one-item  version)  are  counted  as 
unique  measures. 
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Table  3.4 

Measures  with  Clinical  Significance  Scores,  by  Domain 


Domain 

Number  of  Measures  with 
Clinical  Cutoff  Scores 

Number  of  Measures  with 
Clinical  Cutoff  Scores  and  Used 
with  Military  Populations 

PTSD 

24 

17 

Depression 

12 

5 

Anxiety 

10 

5 

TBI 

4 

4 

Suicidal  thoughts 

5 

4 

Stress  and  coping 

2 

2 

Exposure  to  traumatic 
events 

3 

3 

NOTE:  Measures  can  cover  more  than  one  domain;  therefore,  numbers  in  this  table  do  not 
represent  unique  measures  and  the  table  columns  cannot  be  summed.  Multiple  versions  of 
a  single  measure  (e.g.,  a  two-item  version  and  a  one-item  version)  are  counted  as  unique 
measures.  Resiliency,  force  readiness,  and  unit  cohesion  are  excluded  from  this  table 
because  they  are  not  clinical  measures. 


cal  cutoff  scores,  among  which  PTSD  and  depression  are  the  most  common.  We  were  able  to 
identify  only  a  limited  number  of  measures  with  clinical  significance  scores  that  had  been  used 
with  military  populations;  however,  17  measures  of  PTSD  have  clinical  cutoff  scores  and  have 
been  used  with  military  populations. 


Psychometrics 

The  presence  of  published  psychometrics  can  be  an  important  indicator  of  the  extent  to  which 
the  measure  assesses  the  intended  construct,  and  does  so  in  a  manner  that  can  be  repeated 
across  multiple  administrations  or  for  different  types  of  respondents.  When  searching  the 
ROMR  database  for  available  measures,  program  evaluators  may  wish  to  consider  only  mea¬ 
sures  that  have  published  psychometric  information,  as  this  could  help  narrow  a  search  strategy 
to  include  only  the  most  established  measures.  This  is  especially  important  if  the  evidence  gath¬ 
ered  during  program  evaluation  efforts  will  be  used  to  effect  change  within  a  program,  making 
it  essential  that  collected  information  accurately  reflects  the  desired  constructs.  The  ROMR 
includes  specific  information  that  describes  the  reliability  and/or  validity  of  each  instrument 
described  in  published  work,  as  well  as  a  short  primer  to  help  users  interpret  this  information 
(Appendix  G).  Table  3.5  provides  a  summary  of  measures  for  which  information  on  both  the 
reliability  and  validity  of  the  measure  was  available  in  the  identified  literature.  There  are  fewer 
measures  with  published  psychometric  information  that  have  been  validated  with  military 
populations,  and  even  fewer  that  are  available  at  no  cost.  Additional  work  is  needed  to  validate 
many  of  these  measures  in  military  populations,  especially  no-cost  measures  of  anxiety  and 
measures  related  to  TBI. 
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Table  3.5 

Measures  with  Published  Psychometrics,  by  Domain 


Measures 

Number  of  Measures 
Identified 

Number  of  Measures 
Used  with  Military 
Populations 

Number  of  Measures 
Used  with  Military 
Populations  and 
Available  at  No  Cost 

PTSD 

49 

36 

25 

Depression 

71 

31 

20 

Anxiety 

41 

18 

7 

TBI 

20 

13 

4 

Suicidal  thoughts 

16 

7 

3 

Resiliency 

15 

10 

5 

Stress  and  coping 

16 

10 

5 

Exposure  to  traumatic 
events 

21 

15 

10 

Force  readiness 

4 

4 

3 

Unit  cohesion 

10 

10 

7 

NOTE:  Measures  can  cover  more  than  one  domain;  therefore,  numbers  in  this  table  do  not 
represent  unique  measures  and  the  table  columns  cannot  be  summed.  Multiple  versions  of 
a  single  measure  (e.g.,  a  two-item  version  and  a  one-item  version)  are  counted  as  unique 
measures.  Published  psychometrics  includes  data  on  both  the  reliability  and  validity  of  the 
measure. 


CHAPTER  FOUR 


Potential  Uses  of  the  RAND  Online  Measure  Repository 


The  ROMR  has  a  number  of  potential  uses  across  a  wide  variety  of  programs  and  by  diverse 
groups  of  professionals.  This  chapter  provides  more  detail  on  how  the  ROMR  can  be  used  by 
program  evaluators,  researchers,  clinicians,  and  others  interested  in  measuring  psychological 
health  and  constructs  related  to  TBI.  Appendix  H  provides  a  more  detailed  user  guide  with 
specific  instructions  about  how  to  access  and  search  the  ROMR. 


Select  Measures  for  Program  Evaluation  or  Research  Related  to  Psychological 
Health  and  Traumatic  Brain  Injury 

The  primary  purpose  of  the  ROMR  is  to  help  researchers  and  program  evaluators  select  appro¬ 
priate  measures  based  on  the  needs  of  their  program  and  the  population  it  serves.  Researchers, 
program  evaluators,  and  others  can  use  the  repository  to  identify  specific  measures  across  a 
wide  variety  of  domains  related  to  psychological  health,  including  depression,  anxiety,  PTSD, 
stress  and  coping,  and  resiliency.  The  ROMR  also  includes  measures  related  to  TBI.  Besides 
identifying  the  content-area  domain  of  the  measure,  the  ROMR  includes  additional  informa¬ 
tion  to  help  users  select  appropriate  measures  to  meet  their  needs,  such  as 

•  type  of  staff  who  can  administer  the  measure 

•  whether  or  not  the  measure  has  been  used  with  a  military  population 

•  age  group  for  whom  the  measure  is  appropriate  (e.g.,  adults  versus  children) 

•  length  of  a  specific  measure  (number  of  items) 

•  method  of  administration  (e.g.,  questionnaire,  interview) 

•  published  information  on  reliability  and  validity. 


Select  Measures  for  Dual  Use  by  Both  Clinicians  and  Program  Evaluators  or 
Researchers 

Although  the  ROMR  is  primarily  intended  for  use  by  program  evaluators,  it  does  include  some 
measures  that  can  be  used  both  for  research  purposes  and  to  inform  clinical  care.  The  ROMR 
includes  information  on  available  clinical  cutoff  scores  that  are  recommended  for  use  in  the 
peer-reviewed  literature  to  determine  when  individuals  require  clinical  services,  to  inform  clin¬ 
ical  case  planning,  or  to  screen  individuals  who  may  be  at  risk  for  developing  a  psychological 
disorder.  Measures  with  clinical  meaning  (i.e.,  those  with  clinical  cutoff  scores)  may  be  useful 


19 


20  The  RAND  Online  Measure  Repository  for  Evaluating  Psychological  Health  and  Traumatic  Brain  Injury  Programs 


to  clinicians  providing  novel  interventions  or  other  services  specifically  targeting  clinical  out¬ 
comes,  as  well  as  to  researchers  or  evaluators  who  may  be  working  with  these  clinicians  to 
determine  the  effectiveness  of  their  services. 


Identify  Core  Outcome  Measures  for  Evaluating  Similar  Programs 

Government  agencies,  foundations,  or  others  responsible  for  monitoring  a  set  of  programs 
could  use  the  ROMR  to  identify  and  endorse  a  specific  set  of  measures  that  are  both  reliable 
and  valid  across  a  variety  of  domains.  Consistency  in  tracking  core  outcomes  or  indicators  is 
one  important  step  toward  being  able  to  compare  the  effectiveness  of  different  programs  in 
order  to  determine  which  approaches  are  achieving  the  greatest  benefits  for  the  population 
served.  Measures  in  the  ROMR  could  also  be  reviewed  to  identify  a  set  of  screening  and  assess¬ 
ment  instruments  that  could  be  instituted  consistently  across  primary  care  and  mental  health 
settings. 


Determine  Need  for  Additional  Development  or  Validation  of  Measures  with 
Military  Populations 

The  ROMR  contains  a  broad  array  of  measures  that  assess  psychological  health  and  constructs 
related  to  TBI  that  have  been  described  in  the  peer-reviewed  literature  over  the  past  10  years. 
Examining  the  existing  measures  currently  in  use  by  researchers  and  program  evaluators  can 
be  helpful  in  determining  areas  in  need  of  additional  measure  development.  For  example,  there 
are  few  measures  of  force  readiness  currently  being  used  for  research  or  program  evaluation, 
and  the  reliability  and  validity  of  these  measures  has  not  been  described  in  the  published  lit¬ 
erature.  Similarly,  for  some  domains  (e.g.,  anxiety),  few  measures  have  been  validated  for  use 
in  the  military  population.  Further  information  about  the  psychometric  properties  of  these 
measures  will  be  useful  in  determining  their  usefulness  for  these  purposes. 


Conclusions 

The  ROMR  is  a  valuable  tool  that  can  facilitate  the  evaluation  and  monitoring  of  programs 
by  providing  an  online  searchable  database  of  measures  related  to  psychological  health  and 
TBI.  To  maintain  its  usefulness,  the  ROMR  will  require  frequent  updating  to  include  emerg¬ 
ing  measures  and  to  supplement  information  on  the  reliability  and  validity  of  measures  as  it 
is  published.  Additions  may  also  be  considered  to  expand  areas  of  the  ROMR  where  fewer 
measures  have  been  identified.  These  areas  include  measures  of  leadership,  force  readiness,  unit 
cohesion,  and  family  support.  Designated  DoD  organizations  with  responsibility  for  assessing 
the  impact  of  psychological  health  and  TBI  programs  should  consider  making  provisions  to 
update  the  database  regularly  to  ensure  its  continued  relevancy. 


APPENDIX  A 


Existing  Measure  Repositories 


This  appendix  describes  the  existing  measure  repositories  or  databases  that  RAND  reviewed 
when  identifying  the  key  features  and  functions  that  should  be  included  in  the  ROMR.  These 
brief  descriptions  feature  the  number  and  types  of  measures  in  each  repository,  as  well  as  the 
type  of  information  that  each  repository  includes  about  each  measure. 


ADAI  Substance  Use  Assessment  Instrument  Library  (2011) 

The  Alcohol  and  Drug  Abuse  Institute  (ADAI)  at  the  University  of  Washington  has  a  free, 
online  database  of  750  measures  related  to  the  screening  and  assessment  of  substance  use  and 
substance  use  disorders.  Available  information  on  each  measure  includes  the  measure  developer 
and  contact  information,  a  link  to  the  measure,  the  instrument  type  and  target  population, 
scoring  information,  and  validity/reliability  information  on  the  measure  (as  available).  The 
measures  are  searchable  by  the  target  population,  the  measure’s  intended  use  (self-assessment, 
outcome  evaluation,  etc.),  and  keyword.  Veterans  are  a  searchable  target  population,  but  there 
are  only  seven  measures  attributed  to  this  population. 


Buros  Institute  of  Mental  Measurements  (2011) 

The  Buros  Institute  of  Mental  Measurements  reviews  commercially  available  tests  that  have 
psychometric  data.  A  hard  copy  of  the  Mental  Measurements  Yearbook  costs  approximately 
$200.  The  Buros  Institute  also  hosts  a  free,  searchable  online  database  of  more  than  3,500 
tests,  although  only  a  very  limited  amount  of  information  is  available.  Further  detail  is  avail¬ 
able  by  purchasing  a  $15  Buros  review,  which  is  a  more  detailed  report  available  individually 
for  each  of  the  3,500  tests.  These  reviews  contain  basic  information  on  the  measure  (including 
the  developer  and  pricing  information),  a  measure  description,  technical  information  on  the 
measure,  and  Buros  Institute  commentary.  The  actual  measure  itself  is  not  included.  The  Year¬ 
book  divides  its  measures  into  18  major  categories  including  behavior  assessment,  intelligence 
and  general  aptitude,  neuropsychological,  and  speech  and  hearing.  No  measures  related  to  TBI 
are  included. 


21 


22  The  RAND  Online  Measure  Repository  for  Evaluating  Psychological  Health  and  Traumatic  Brain  Injury  Programs 


Center  for  Outcome  Measurement  in  Brain  Injury  (2011) 

The  Center  for  Outcome  Measurement  in  Brain  Injury  is  an  online  resource  for  individuals 
needing  detailed  information  and  support  in  using  outcome  measures  related  to  brain  injuries. 
The  website  contains  31  measures  that  are  commonly  used  in  the  field  of  brain  injury  reha¬ 
bilitation  and  assessment.  Available  information  on  each  measure  includes  a  brief  description, 
measure  reference,  and  how  to  access  the  measure. 


Compendium  of  Assessment  and  Research  Tools  (CART)  (2011) 

The  Compendium  of  Assessment  and  Research  Tools  (CART),  supported  by  the  Star  Center 
and  W.  K.  Kellogg  Foundation,  is  an  online  database  of  nearly  700  instruments  for  youth 
development  programs.  Use  of  the  database  is  free;  it  is  searchable  by  domains  and  subdo¬ 
mains,  such  as  design  and  implementation  (basic  program  elements),  context  (factors  that  have 
an  indirect  effect  on  implementation),  and  outcomes  (where  a  change  is  expected  to  occur). 
Available  information  on  each  measure  includes  (as  applicable  and  available  for  each  measure) 
the  target  population,  year  created  or  modified,  instrument  type,  a  short  description,  a  link  to 
the  measure  itself,  and  associated  constructs.  This  database  does  not  have  measures  related  to 
military  service  members. 


ETS  Test  Collection/Carl  C.  Brigham  Library  (2011) 

The  Educational  Testing  Service  has  more  than  25,000  items  in  its  database.  Measures  in  the 
ETS  database  cover  a  variety  of  domains  including  general  health  and  well-being,  and  educa¬ 
tional  and  career  achievement.  The  information  in  the  database  is  designed  for  use  by  research¬ 
ers,  graduate  students,  and  teachers.  There  are  no  specific  references  to  a  military  population. 
More  than  1,000  tests  are  downloadable  for  a  $25  distribution  fee  (abstracts  can  be  viewed 
free).  As  of  February  2,  2011,  38  of  these  measures  were  related  to  PTSD  (of  which  7  items 
were  related  to  “combat”),  and  14  measures  were  related  to  TBI. 


Measuring  Violence-Related  Attitudes,  Behaviors,  and  Influences  Among 
Youths:  A  Compendium  of  Assessment  Tools,  2nd  ed.  (2011) 

This  compendium  published  by  the  Centers  for  Disease  Control,  National  Center  for  Injury 
Prevention  and  Control,  provides  researchers  and  prevention  specialists  with  a  free  set  of  tools 
to  assess  violence-related  beliefs,  behaviors,  and  influences,  as  well  as  to  evaluate  programs 
to  prevent  youth  violence.  There  are  more  than  170  measures  intended  for  use  with  youths 
between  the  ages  of  1 1  and  24  years  to  assess  serious  violent  and  delinquent  behavior,  conflict 
resolution  strategies,  social  and  emotional  competencies,  peer  influences,  parental  monitoring 
and  supervision,  family  relationships,  exposure  to  violence,  collective  efficacy,  and  neighbor¬ 
hood  characteristics.  Available  information  on  each  measure  includes  a  brief  description,  reli¬ 
ability  and  validity,  target  population,  and  the  measure  developer. 
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National  Center  for  the  Study  of  PTSD  (2011) 

The  National  Center  for  the  Study  of  PTSD  is  a  federal  research  and  education  agency  within 
the  Department  of  Veterans  Affairs.  It  has  developed  a  52-measure  database  that  is  free  to 
use  and  includes  measures  relevant  to  PTSD,  including  both  trauma  exposure  measures  and 
PTSD  screening  tools  for  adults  and  children.  Available  information  on  each  measure  includes 
a  brief  description,  sample  item,  reference,  and  information  about  how  to  obtain  the  measure. 


Patient  Reported  Outcome  and  Quality  of  Life  Instruments  Database 
(ProQolid)  (2011) 

The  Patient  Reported  Outcome  and  Quality  of  Life  Instruments  Database  (ProQolid)  is  an 
online  database  of  nearly  700  instruments  developed  by  the  MAPI  Research  Institute.  The 
database  has  two  levels:  free  and  advanced  (for  a  fee).  The  free  level  offers  14  data  fields  of  infor¬ 
mation  on  an  instrument  including  purpose,  population,  and  some  basic  characteristics  (such 
as  the  type  of  instrument).  The  advanced  level  contains  psychometric  properties  of  measures, 
methodology  of  instrument  development,  and  the  final  version  of  the  measure;  access  costs  a 
minimum  of  $750.  The  measure  domains  are  related  more  to  physical  ailments  than  to  mental 
health,  although  there  are  measures  related  to  psychiatry  and  psychology,  which  include  anxi¬ 
ety  and  depression  measures.  Military  service  members  are  not  a  specific  population  targeted 
by  this  database. 


APPENDIX  B 


Detailed  Literature  Search  Strategies  Used  to  Identify  Measures 


Table  B.1 


Database  Domain  Limits 


Search  Concepts 


Results 


Military 

Depression,  2008  to 

Psychology 

anxiety,  current 

PTSD, 

resilience. 

and  suicide 

Military  TBI 

2008  to 

Psychology 

current 

Concept  1:  depression  OR  anxiety  OR  posttraumatic 
stress  OR  PTSD  OR  resilience  OR  suicide  OR  suicidal 
AND 

Concept  2:  measure  OR  inventory  OR  scale  OR 
assessment  OR  psychometrics  OR  questionnaire  OR 
screening  OR  survey  OR  checklist  OR  test 
AND 

Concept  3:  journal  title:  Military  Psychology 

Concept  1:  traumatic  brain  injury 
AND 

Concept  2:  survey  OR  instrument  OR  questionnaire 
OR  scale  OR  measure  OR  psychometrics  OR 
screening  OR  test  OR  span  OR  retention  OR 
psychomotor  OR  sociometric  OR  checklist 
AND 

Concept  3:  journal  title:  Military  Psychology 


PsycINFO 


Depression, 
anxiety, 
PTSD, 
resilience, 
and  suicide 


English, 
published 
in  2000  to 
current,  peer- 
reviewed 
journal 
articles  only 


Concept  1:  attitude  measure*  OR  inventory  OR 
inventories  OR  multidimensional  scaling  OR 
psychiatric  evaluation  OR  psychological  assessments 
OR  psychometrics  OR  questionnaires  OR  rating 
scales  OR  screening  OR  statistical  measurement 
OR  subtests  OR  surveys  OR  symptom  checklists  OR 
testing 
AND 

Concept  2:  posttraumatic  stress  disorder  OR  DE 
"acute  stress  disorder"  or  DE  "death  anxiety"  or 
DE  "generalized  anxiety  disorder"  or  DE  "obsessive 
compulsive  disorder"  or  DE  "panic  disorder"  or  DE 
"phobias"  or  Tl  anxiety  OR  DE  "dysthymic  disorder" 
or  DE  "endogenous  depression"  or  DE  "reactive 
depression"  or  DE  "recurrent  depression"  or  DE 
"treatment  resistant  depression"  OR  Tl  depression 
OR  resilience  OR  suicid* 

AND 

Concept  3:  Tl  valid*  OR  DE  valid*  OR  Tl  reliab*  OR 
DE  reliab* 


54  total, 

1  duplicate, 

2  possibly 
relevant 


6  total, 

4  duplicates, 
none  relevant 


790  total; 
after  deleting 
duplicates,  309 
unique  results, 
with  95  judged 
possibly 
relevant 
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Table  B.1 — Continued 


Database  Domain  Limits 


Search  Concepts 


Results 


PubMed  Anxiety,  Humans, 
depression,  English, 
and  PTSD  published 
in  2000  to 
current 


Concept  1:  posttraumatic  stress  disorder*  OR  829  total, 

depressive  disorder[TI]  OR  depression[MeSH  Terms]  with 
OR  depression[TI]  OR  anxiety[MeSH  Terms]  OR  257  judged 

anxiety[TI]  OR  depressive  disorder,  major[MeSH]  OR  possibly 

dysthymic  disorder[Mesh]  relevant 

AND 

Concept  2:  survey  OR  surveys  OR  instrument  OR 


instruments  OR  questionnaire  OR  questionnaires  OR 
psychiatric  status  rating  scales  OR  measure[tiab]  OR 
measures[tiab]  OR  psychiatric  status  rating  scales  OR 
psychometrics 
AND 

Concept  3:  validation  studies  as  topic[Mesh]  OR 
valid*[ti]  OR  reliab*[ti] 

NOT 

Concept  4:  postpartum  OR  post-partum  OR  post¬ 
natal  OR  postnatal 
NOT 

Concept  5:  editorial  OR  letter  OR  comment 


PILOTS  All 


English, 
published 
in  2000  to 
current,  peer- 
reviewed 
journal 
articles  only 


Concept  1:  survey  OR  surveys  OR  instrument  OR 
instruments  OR  questionnaire  OR  questionnaires 
OR  psychiatric  status  rating  scales  OR  measure  OR 
measures  OR  psychiatric  status  rating  scales  OR 
psychometrics 
AND 

Concept  2:  valid*  OR  reliab* 


674  total; 
after  deleting 
duplicates,  601 
unique  results, 
with  141  results 
judged  possibly 
relevant 


PsycINFO  TBI 


English, 
published 
in  2000  to 
current,  peer- 
reviewed 
journal 
articles  only 


Concept  1:  aptitude  measure*  OR  attitude 
measure*  OR  comprehension  test*  OR  digit  span 
test*  OR  intelligence  measure*  OR  inventory 
OR  inventories  OR  multidimensional  scaling  OR 
perceptual  measure*  OR  psychiatric  evaluation 
OR  psychological  assessments  OR  psychometrics 
OR  questionnaires  OR  rating  scales  OR  retention 
measures  OR  screening  OR  sensorimotor  measures 
OR  sociometric  test*  OR  speech  and  hearing 
measures  OR  statistical  measurement  OR  subtests 
OR  survey*  OR  symptom  checklist*  OR  testing  OR 
verbal  test* 

AND 

Concept  2:  traumatic  brain  injur* 

AND 

Concept  3:  Tl  valid*  OR  DE  valid*  OR  Tl  reliab*  OR 
DE  reliab* 


116  total; 
after  deleting 
duplicates,  94 
unique  results, 
with  76  judged 
possibly 
relevant 


PubMed  TBI 


Humans, 
English, 
published 
in  2000  to 
current 


Concept  1:  traumatic  brain  injury 
AND 

Concept  2:  survey  OR  surveys  OR  instrument  OR 
instruments  OR  questionnaire  OR  questionnaires 
OR  psychiatric  status  rating  scales  OR  measure[tiab] 
OR  measures[tiab]  OR  psychiatric  status  rating 
scales  OR  psychometrics  OR  aptitude  tests  OR 
digit  span  OR  language  tests  OR  hearing  tests  OR 
neuropsychological  tests  OR  retention  [MeSH] 

OR  psychomotor  performance  OR  sociometric 
techniques  OR  speech  production  measurement 
AND 

Concept  3:  "validation  studies  as  topic"[MeSH]  OR 
valid*[tiab]  OR  reliab*[tiab] 

NOT 

Concept  4:  editorial  OR  letter  OR  comment 


89  total, 

1  duplicate, 
with  64  unique 
results  judged 
possibly 
relevant 
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Table  B.1 — Continued 


Database  Domain  Limits 


Search  Concepts 


Results 


PubMed  Resilience  Humans, 
and  suicidal  English, 
thoughts  published 
in  2000  to 
current 


Concept  1:  resilienc*  OR  suicid* 

AND 

Concept  2:  survey  OR  surveys  OR  instrument  OR 
instruments  OR  questionnaire  OR  questionnaires  OR 
psychiatric  status  rating  scales  OR  measure[tiab]  OR 
measures[tiab]  OR  psychiatric  status  rating  scales  OR 
psychometrics 
AND 

Concept  3:  "validation  studies  as  topic"[MeSH]  OR 
valid*[ti]  OR  reliab*[ti] 

NOT 

Concept  4:  editorial  OR  letter  OR  comment 


94  total; 
after  deleting 
duplicates,  73 
unique  results, 
with  14  judged 
possibly 
relevant 


PILOTS 


Military 

English, 

Concept  1:  survey  OR  surveys  OR  instrument  OR  1  total 

measures 

published 

instruments  OR  questionnaire  OR  questionnaires 

in  2000  to 

OR  psychiatric  status  rating  scales  OR  measure  OR 

current,  peer- 

measures  OR  psychiatric  status  rating  scales  OR 

reviewed 

psychometrics 

journal 

AND 

articles  only 

Concept  2:  valid*  OR  reliab* 

AND 

military  measures  concepts:  military  AND  readiness, 

"unit  cohesion,"  "force  preservation,"  military  AND 
(confidence  OR  trust  OR  perception*  OR  support*) 

AND  (leader*  OR  unit) 

PsycINFO  Military 
measures 


English, 
published 
in  2000  to 
current,  peer- 
reviewed 
journal 
articles  only 


Concept  1:  attitude  measure*  OR  inventory  OR 
inventories  OR  multidimensional  scaling  OR 
psychiatric  evaluation  OR  psychological  assessments 
OR  psychometrics  OR  questionnaires  OR  rating 
scales  OR  screening  OR  statistical  measurement 
OR  subtests  OR  surveys  OR  symptom  checklists  OR 
testing 
AND 

Concept  2:  valid*  OR  reliab* 

AND 

military  measures  concepts:  military  AND  readiness, 
"unit  cohesion,"  "force  preservation,"  military  AND 
(confidence  OR  trust  OR  perception*  OR  support*) 
AND  (leader*  OR  unit) 


17  total, 

5  duplicates 


PubMed  Military  Humans, 
measures  English, 
published 
in  2000  to 
current 


Concept  1:  survey  OR  surveys  OR  instrument  OR  23  total, 

instruments  OR  questionnaire  OR  questionnaires  OR  2  duplicates 
psychiatric  status  rating  scales  OR  measurejtiab]  OR 
measuresjtiab]  OR  psychiatric  status  rating  scales  OR 
psychometrics 
AND 

Concept  2:  "validation  studies  as  topic"[MeSH]  OR 
valid*[tiab]  OR  reliab*[tiab] 

AND 

military  measures  concepts:  military  AND  readiness, 

"unit  cohesion,"  "force  preservation,"  military  AND 
(confidence  OR  trust  OR  perception*  OR  support*) 

AND  (leader*  OR  unit) 
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Acute  Stress  Checklist  for  Children  (ASC-Kids) 

Adolescent  Cognitive  Style  Questionnaire  (ACSQ) 

Adult  Suicidal  Ideation  Questionnaire  (ASIQ) 

Anxiety  and  Depression  Detector  (ADD) 

Anxiety  Disorders  Interview  Schedule  (ADIS) 

Anxiety  Sensitivity  Index-3  (ASI-3) 

Anxiety  Sensitivity  Profile-22  Item  Version  (ASP-22) 

Appraisal  of  Social  Concerns  Scale  (ASC) 

Army  Post-Deployment  Reintegration  Scale  (APDRS) 

Assessment  of  Depression  Inventory  (ADI) 

Beck  Hopelessness  Scale  (BHS) 

Beck  Scale  for  Suicide  Ideation  (SSI) 

Behavior  and  Symptom  Identification  Scale  (BASIS-32) 

Behavioral  Observation  System  (BOS) 

Bipolar  Inventory  of  Symptoms  Scale  (BISS) 

Brief  Measures  of  Positive  and  Negative  Affect  (PANAS) 

Brief  Traumatic  Brain  Injury  Screen  (BTBIS) 

California  Psychological  Inventory-Depression  (CPI-D) 

Center  for  Epidemiologic  Studies  Depression  Scale  (CES-D) 

Changes  in  Outlook  Questionnaire  (CiOQ) 

Changes  in  Outlook  Questionnaire-Short  Form  (CiOQ-SF) 

Child  Health  and  Illness  Profile— Child  Edition,  Child  Report  Form  (CHIP-CE/CRF) 
Child  Health  and  Illness  Profile— Child  Edition,  Parent  Report  Form  (CHIP-CE/PRF) 
Childhood  Trauma  Questionnaire  (CTQ) 

Children’s  Revised  Impact  of  Event  Scale-8  Item  Version  (CRIES-8) 

Children’s  Revised  Impact  of  Event  Scale-13  Item  Version  (CRIES-13) 

Clinically  Useful  Depression  Outcome  Scale  (CUDOS) 

Clinician-Administered  PTSD  Scale  (CAPS) 

CNS  Vital  Signs  Battery  (CNSVS) 

Cognitive  Style  Questionnaire  (CSQ) 

Cohesion  Questionnaire-Brigade  Survey  (CQ-BS) 

Cohesion  Questionnaire-Company  Survey  (CQ-CS) 

Combat  Exposure  Scale  (CES) 

Composite  International  Diagnostic  Interview  (CIDI)  -  PTSD  Module 
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Concussion  Resolution  Index  (CRI) 

Connor-Davidson  Resilience  Scale  (CD-RISC)  -  10  Item  Version 
Connor-Davidson  Resilience  Scale  (CD-RISC)  -  25  Item  Version 
Connor-Davidson  Resilience  Scale  (CD-RISC2)  -  2  Item  Version 
Cornell  Dysthymia  Rating  Scale  (CDRS) 

Davidson  Trauma  Scale  (DTS) 

Depersonalization  Severity  Scale  (DSS) 

Deployment  Risk  and  Resilience  Inventory  (DRRI) 

Depression  Arkansas  Scale  (D-ARK) 

Depression  Coping  Self-Efficacy  Scale  (DCSES) 

Disability  Rating  Scale  (DRS) 

Dispositional  Resilience  Scale-15  Item  Version  (DRS15) 

Distressing  Event  Questionnaire  1  (DEQ  1) 

Distressing  Event  Questionnaire  2  (DEQ  2) 

Emotional  Reactivity  and  Numbing  Scale  (ERNS) 

Fear  of  Negative  Evaluation-Brief  Version  (FNEB) 

General  Behavior  Inventory-Parent  Report  (GBI-P) 

Generalized  Anxiety  Disorder  Severity  Scale  (GADSS) 

Global  Assessment  Scale  (GAS) 

Global  Assessment  Tool  (GAT) 

Hamilton  Rating  of  Depression  Scale/Montgomery  Asberg  Depression  Rating  Scale  Interview 
(HMI) 

Hamilton  Rating  Scale  for  Anxiety  (HAMA) 

Hamilton  Rating  Scale  for  Depression  (HAMD) 

Impact  of  Event  Scale  (IES) 

Impact  of  Event  Scale-Revised  (IES-R) 

Impact  of  Event  Scale-6  Item  Version  (IES-6) 

Injury  Distress  Index  (IDI) 

Integrated  Delivery  System  Consultation  Assessment  Tool  (IDS-CAT) 

Interactive  Voice  Response  Montgomery  Asberg  Depression  Rating  Scale  (IRV  MADRS) 
Interview  for  Mood  and  Anxiety  Symptoms  (IMAS) 

Inventory  of  Depression  and  Anxiety  Symptoms  (IDAS) 

Inventory  of  Depression  and  Anxiety  Symptoms— Clinical  Rating  (IDAS-CR) 

Inventory  of  Depressive  Symptomatology,  Clinician  Rating  (IDS-C) 

Inventory  of  Depressive  Symptomatology,  Self  Report  (IDS-SR) 

Joint  Mental  Health  Advisory  Team  7  Survey  (J-MHAT  7) 

Joint  Mental  Health  Advisory  Team  7  Survey  (J-MHAT  7)  -  Barriers  and  Stigma  Scale 
Liebowitz  Social  Anxiety  Scale  (LSAS) 

Life  Satisfaction  Index-A  (LSI) 

Lifetime  Trauma  and  Victimization  History  (LTVH) 

Los  Angeles  Symptom  Checklist  (LASC) 

Mental  Health  Inventory  (MHI) 

Military  Life  Scale  (MLS) 

Minnesota  Multiphasic  Personality  Inventory— 2  (MMPI-2) 

Mississippi  PTSD  Scale— Revised  (M-PTSD  R) 

Mississippi  Scale  for  Combat-Related  PTSD  (M-PTSD) 
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Modified  Scale  for  Suicidal  Ideation  (MSSI) 

Montgomery  Asberg  Depression  Rating  Scale  (MADRS) 

Mood  and  Anxiety  Symptoms  Questionnaire  (MASQ) 

Mood  and  Feelings  Questionnaire-Child  (MFQ-C) 

Mood  and  Feelings  Questionnaire— Parent  Version  (MFQ-P) 

Mood  and  Feelings  Questionnaire-Single  Item  Child  Version  (MFQ-C1) 

Mood  and  Feelings  Questionnaire— Single  Item  Parent  Version  (MFQ-P1) 

Mood  and  Feelings  Questionnaire— 2  Item  Child  Version  (MFQ-C2) 

Mood  and  Feelings  Questionnaire— 2  Item  Parent  Version  (MFQ-P2) 

Moss  Attention  Rating  Scale  (MARS) 

My  Mood  Monitor  Checklist  (M-3) 

National  Anxiety  Disorder  Screening  Day  Instrument  (NADSD) 

Neurological  Outcome  Scale  for  Traumatic  Brain  Injury  (NOS-TBI) 

Neuropsych  Questionnaire  (NPQ)  -  Adult  Version 
Neuropsych  Questionnaire- Short  Form  (NPQ-SF) 

Neuropsychological  Assessment  Battery-Screening  Module  (NAB-SM) 
Non-Traumatic  Stressors  Questionnaire  (NTSQ) 

Numeric  Anxiety  Scale  (NAS) 

Patient  Health  Questionnaire  (PHQ-8) 

Patient  Health  Questionnaire-9  Item  Version  (PHQ-9) 

Patient  Health  Questionnaire-2  Item  Version  (PHQ-2) 

Pediatric  Inpatient  Behavior  Scale  (PIBS) 

Pediatric  Quality  of  Life  Enjoyment  and  Satisfaction  Questionnaire  (PQ-LES-Q) 
Penn  Inventory  for  Posttraumatic  Stress  Disorder  (Penn  Inventory) 

Pentagon  Post  Disaster  Health  Assessment  (PPDHA) 

Peritraumatic  Dissociative  Experiences  Questionnaire  (PDEQ) 

Peritraumatic  Distress  Inventory  (PDI) 

Personality  Assessment  Inventory  (PAI) 

Pittsburgh  Sleep  Quality  Index  Addendum  for  PTSD  (PSQI-A) 

Positive  and  Negative  Suicide  Ideation  Inventory  (PANSI) 

Post-Deployment  Readjustment  Inventory  (PDRI) 

Posttraumatic  Growth  Inventory  (PTGI) 

Posttraumatic  Stress  Diagnostic  Scale  (PSDS) 

Primary  Care-PTSD  Screen  (PC-PTSD) 

Psychiatric  Diagnostic  Screening  Questionnaire  (PDSQ) 

PTSD  Checklist  (PCL) 

PTSD  Checklist— 2  Item  Version  (PCL-2) 

PTSD  Checklist-3  Item  Version  (PCL-3) 

PTSD  Checklist-4  Item  Version  (PCL-4) 

PTSD  Checklist— 6  Item  Version  (PCL-6) 

PTSD  Symptom  Scale-Interview  Version  (PSS-I) 

Purdue  Posttraumatic  Stress  Disorder  Scale,  Revised  (PPTSD-R) 

Quick  Inventory  of  Depressive  Symptomatology,  Clinician  Rating  (QIDS-C) 
Quick  Inventory  of  Depressive  Symptomatology,  Self-Report  (QIDS-SR) 

RAND  Peritraumatic  Dissociative  Experiences  Questionnaire  (RAND  PDEQ) 
Readiness  Estimate  and  Deployability  Index  (READI) 
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Readiness  Estimate  and  Deployability  Index  Revised  for  Air  Force  Nurses  (READI-R-AFRN) 
Readiness  Estimate  and  Deployability  Index  Revised  for  Air  Force  Nurses  Short  Form 
(READI-R-AFRN  [SF]) 

Repeatable  Battery  for  the  Assessment  of  Neurological  Status  (RBANS) 

Revised  Child  Anxiety  and  Depression  Scale  (RCADS) 

Screen  for  Child  Anxiety- Related  Emotional  Disorders  (SCARED) 

Screen  for  Posttraumatic  Stress  Symptoms  (SPTSS) 

Self-Harm  and  Behavior  Questionnaire  (SHBQ) 

Self-Injurious  Thoughts  and  Behavior  Interview  (SITBI) 

Short-Form  Health  Survey-12  item  (SF-12) 

Short  Form  Health  Survey-36  Item  (SF-36) 

Short  Mood  and  Feelings  Questionnaire-Child  Version  (SMFQ-C) 

Short  Mood  and  Feelings  Questionnaire-Parent  Version  (SMFQ-P) 

Short  Screening  Scale  for  PTSD 

Single-Item  Global  Measures  of  the  Severity  of  Depression  (GSEVDEP) 

Single-Item  Global  Psychological  Functioning  Questionnaire  (CPF) 

Single-Item  Global  Quality  of  Life  Questionnaire  (GQOL) 

Social  Anxiety  Scale  for  Adolescents  (SAS-A) 

Social  Anxiety  Scale  for  Adolescents  Revised  (SAS-AR) 

Social  Knowledge  Test  (SKT) 

Social  Phobia  and  Anxiety  Inventory  (SPAI) 

Social  Phobia  and  Anxiety  Inventory  for  Children  (SPAI-C) 

State-Trait  Anxiety  Inventory  (STAI) 

Stressful  Life  Events  Schedule-Child  Version  (SLES-C) 

Structured  Clinical  Interview  for  DSM-IV  Dissociative  Disorders-Revised  (SCID-D-R) 
Structured  Clinical  Interview  for  DSM-IV  PTSD  Screening  Module  (SCID-PTSD) 
Structured  Interview  for  Disorders  of  Extreme  Stress-Revised  (SIDES-R) 

Structured  Interview  Guide  for  the  Hamilton  Anxiety  Scale  (SIGH-A) 

Structured  Interview  Guide  for  the  Montgomery  Asberg  Depression  Rating  Scale  (SIGMA) 
Suicidal  Behaviors  Questionnaire-Revised  (SBQ-R) 

Suicidal  Behaviors  Questionnaire-Single  hem  (SBQ  -1) 

Suicide  Attempt  Self-Injury  Interview  (SASII) 

Symptom  Checklist— 6  hem  Version  (SCL-6) 

Symptom  Checklist— 10  hem  Version  (SCL-10) 

Symptom  Checklist-10  hem  Version,  Revised  (SCL-10R) 

Symptom  Checklist-9 0-Revised  PTSD  Subscale  (SCL-90-R) 

Trails  Making  Test  A  and  B  (TMT) 

Trauma  Symptom  Inventory  (TSI) 

Traumatic  Events  Questionnaire  (TEQ) 

Traumatic  Life  Events  Questionnaire  2  (TLEQ  2) 

Traumatic  Life  Events  Questionnaire  3  (TLEQ  3) 

Type  D  Scale-14  (DS14) 

Veterans  RAND  12  Item  Health  Survey  (VR-12) 

Veterans  RAND  36  Item  Health  Survey  (VR-36) 

Wechsler  Adult  Intelligence  Scale-Ill  Block  Design  (WAIS-III/BD7) 

Wechsler  Adult  Intelligence  Scale— III  Matrix  Reasoning  (WAIS-III/MR7) 
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Wechsler  Adult  Intelligence  Scale  IV  (WAIS-IV) 

Wechsler  Memory  Scale  IV  (WMS-IV) 

Wisconsin  Card  Sorting  Test  (WCST) 

Youth  Risk  Behavior  Survey-2005  Middle  School  Version  (MSYRBS) 


APPENDIX  E 


Data  Abstraction  Form 


Table  E.1 


Elements 
Abstracted 
from  Each 

Article  Brief  Description  of  Each  Element  Response  Options 


General  Information  on  the  Measure 


Name  of  The  full  or  official  name  of  the  measure  Name  and  abbreviation 

measure 


Reference(s)  The  full  reference  information  for  each  article  on  this 
measure.  This  could  include  multiple  references. 


References  and  a  checkbox  for  "no 
other  supporting  articles" 


Brief  summary  One  to  two  sentences  describing  (1)  the  number  of 

of  the  items  in  the  measure,  (2)  how  it  is  administered,  and  (3) 

purpose  of  the  what  it  is  trying  to  measure 
measure 


Brief  summary 


Measure 

domain 


A  list  of  the  following  domains  being  captured  by  the  Multiple  choice  from  the  list  of 
measure:  domains;  more  than  one  domain 

1.  PTSD  could  be  checked  for  each  measure 

2.  Depression 

3.  Anxiety 

4.  TBI 

5.  Suicidal  thoughts 

6.  Resiliency 

7.  Stress  and  coping,  which  includes  measures  of 
general  stress  and  coping  styles  or  skills 

8.  Exposure  to  traumatic  events 

9.  Force  readiness,  which  includes  measures  that  assess 
whether  service  members  are  psychologically  ready 
to  be  deployed 

10.  Unit  cohesion. 


Domains  reflect  areas  captured  by  scales  or  stand-alone 
subscales,  not  single  items. 


Age  group(s)  The  age  group  that  the  measure  was  tested  with: 

1.  Adults  (18  years  and  older) 

2.  Adolescents  or  children. 

If  the  measure  was  used  with  adolescents  or  children, 
the  mean  or  range  of  ages  (e.g.,  4-6  years)  was 
included. 


Multiple  choice  from  list.  Measures 
could  be  applicable  to  both  adults 
and  adolescents  or  children.  There 
were  also  two  integer  fields,  one 
minimum  and  one  maximum  to 
capture  age  ranges. 


Used  with  Whether  or  not  one  of  the  articles  about  this  measure  Yes  or  no 

a  military  shows  that  the  measure  has  ever  been  used  with  any 

population?  military  population. 
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Table  E.1 — Continued 


Elements 
Abstracted 
from  Each 

Article  Brief  Description  of  Each  Element 


Response  Options 


Measure  Administration 

Method  of  A  description  of  how  the  measure  is  administered: 

administration  1.  Survey  questionnaire  (pen  and  paper),  includes 
questionnaires,  inventories,  and  checklists 

2.  Survey  questionnaire  (online) 

3.  Structured  interview 

4.  Semi-structured  interview 

5.  Observation  rating  form 

6.  Other  (please  describe) 

7.  No  information  available. 


Multiple  choice  from  the  list  of 
methods;  more  than  one  method 
could  be  selected  for  a  single 
measure 


Respondent 


A  description  of  who  answers  the  questions  in  the  Multiple  choice  from  the  list  of 

measure:  populations 

1.  Self-administered 

2.  Caregiver  completing  information  about  their  child 

3.  Child  self-report 

4.  Other  (please  describe) 

5.  No  information  available. 


Scoring  the  Measure 


Clinical  cutoff  Whether  or  not  the  measure  has  a  clinical  cutoff  score, 
score  and  what  the  clinical  cutoff  score(s)  is. 

Number  of  The  total  number  of  items  on  the  measure 

items 


Measure  Whether  the  measure  has  subscales,  and  a  list  and  brief 

subscales  description  of  each,  including  the  number  of  items  per 

subscale.  A  subscale  is  a  grouping  on  individual  items 
that  when  considered  together  provide  information 
about  the  same  characteristic. 


Response 
options  and 
anchors 


One  or  two  sentences  that  describe  the  response 
options  and  anchors  for  each  scale  that  comprises  the 
measure  (e.g.,  1  [not  at  all]  to  7  [very  much]).  If  there 
are  multiple  scales  or  subscales,  the  response  options 
and  anchors  for  each  scale  are  described. 


Reliability  and  Validity  of  the  Measure 


Sample  size  (n)  The  range  of  sample  sizes  included  in  the  different 
studies  regarding  this  measure  from  the  smallest  to 
the  largest.  If  one  population  has  been  studied,  only  a 
single  sample  size  is  included. 


Reliability  The  range  of  reliability  scores  on  the  measure  (from 

testing  lowest  to  highest)  across  all  studies  included  in  the 

repository.  Reliability  was  commonly  reported  as 
Cronbach's  alpha,  test-retest  reliability,  or  internal 
consistency. 


Validity  The  overall  correlation  scores  between  this  and  other 

testing  measures.  Validity  was  commonly  established  by 

correlating  the  measure  with  other  similar  measures 
(divergent  validity).  For  screening  and  assessment 
measures,  validity  was  also  established  by  examining 
sensitivity  and  specificity  (determining  if  the  measure 
identifies  individuals  it  is  supposed  to  identify  without 
identifying  individuals  it  is  not  supposed  to  identify). 


Yes  or  no,  and  a  description  of  the 
clinical  cutoff  score(s),  if  available 

Integer  field;  a  "no  information 
available"  option 

Yes  or  no,  and  a  description  of  sub¬ 
scales  (if  applicable) 


A  description  of  response  options 
and  anchors,  and  a  "no  information 
available"  option 


Two  integer  fields,  one  minimum 
and  one  maximum  to  capture 
range,  and  a  "no  information 
available"  option 

A  description  of  reliability  and  a 
"no  information  available"  option 


A  description  of  validity  and  a  "no 
information  available"  option 
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Table  E.1 — Continued 


Elements 
Abstracted 
from  Each 
Article 

Brief  Description  of  Each  Element 

Response  Options 

Locating  the  Measure 

How  to  obtain 
a  copy  of  the 
measure 

A  description  of  how  potential  users  can  obtain  a  copy 
of  the  measure.  For  example,  a  complete  copy  of  the 
measure  may  be  available  as  published  in  one  of  the 
references,  upon  request  from  a  specific  publisher 
or  available  for  download  from  a  specific  website.  If 
measures  were  available  on  a  specific  website,  a  link  to 
the  actual  measure  was  included. 

A  description  of  how  to  obtain  the 
measure 

Fees  to  use 
the  measure 

Whether  there  was  a  fee  to  use  the  measure,  and  a 
description  of  how  much  it  costs  to  use  the  measure 
(e.g.,  $5/per  person),  if  applicable. 

Yes  or  no,  and  a  description  of  the 
cost  of  the  measure,  if  applicable 

Measure  Background 

Developer  of 
the  measure 

A  description  of  who  developed  the  measure,  including 
the  original  reference  to  the  earliest  publication  of  the 
measure  we  could  locate. 

A  reference  and  a  "no  information 
available"  option 

Year  the 

measure  was 

originally 

developed 

The  year  the  measure  was  first  released  for  use. 

An  integer  box  for  the  year  and  a 
"no  information  available"  option 

History  of 
measure 

A  brief  description  of  previous  versions  of  the  measure 
that  were  referenced  in  the  reviewed  articles.  If  this  is 
the  first  version  of  the  measure,  it  is  specified  here. 

A  description  of  previous  versions 
of  the  measures  and  a  "no 
information  available"  option 

Related 

measures 

The  names  of  other  versions  of  the  measure  (e.g., 
short  form,  additional  language  forms)  included  in 
the  measure  repository.  If  there  are  no  other  related 
measures  in  the  measure  repository,  it  is  specified  here. 

A  description  of  other  versions  of 
the  measure  in  the  repository  and  a 
"no  information  available"  option 

APPENDIX  F 


Glossary 


Military  Terms 

force  health  readiness 

The  ability  of  United  States  military  forces  to  optimize  and  protect  the  psychological  and 
physical  health  of  service  members  and  their  families  through  policies  and  programs  across  all 
phases  of  deployment.  It  is  operationalized  through  a  partnership  between  the  service  mem¬ 
bers  who  make  up  the  force,  their  leaders  at  all  levels,  and  health  care  planners  and  providers. 
Force  health  protection  and  readiness  is  also  referred  to  as  force  protection,  force  preservation, 
or  force  preservation  and  readiness.  (Office  of  the  Deputy  Assistant  Secretary  of  Defense,  2011) 

unit  cohesion 

The  bonding  together  of  members  of  an  organization  in  such  a  way  as  to  sustain  their  will  and 
commitment  to  each  other,  their  unit,  and  the  mission.  (Powell  et  al.,  2006) 


Clinical  Terms 

clinical  assessment 

A  measure  administered  by  a  clinician  or  trained  professional  for  the  purpose  of  providing  a 
diagnosis  or  informing  a  treatment  plan. 

screening 

A  measure  used  to  identify  at-risk  populations. 

self-administered  measure 

A  subject  provides  information  about  himself  or  herself,  such  as  on  a  questionnaire  or  in  an 
interview.  (Leary,  1995) 

surveillance 

A  type  of  measure  used  to  determine  the  prevalence  of  a  problem  in  a  given  population  (e.g.,  a 
measure  used  to  determine  the  proportion  of  service  members  affected  by  depression). 
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Research  Method  Terms 

reliability 

The  consistency  or  dependability  of  a  measurement  procedure  or  the  extent  to  which  a  mea¬ 
sured  variable  is  free  from  random  error.  (Leary,  1995;  Stangor,  2004;  Pelham  and  Blanton, 
2007) 

validity 

The  extent  to  which  a  measurement  procedure  actually  measures  what  it  is  intended  to  mea¬ 
sure.  (Leary,  1995) 


APPENDIX  G 


Brief  Guide  to  Interpreting  Measure  Reliability  and  Validity  in  the 
RAND  Online  Measure  Repository 


When  selecting  an  appropriate  measure,  ir  is  critical  to  understand  whether  that  measure  is 
both  reliable  and  valid  for  the  population  you  intend  to  use  it  with.  It  is  important  to  remem¬ 
ber  that  reliability  and  validity  information  in  the  ROMR  is  only  for  the  population  described 
in  the  ROMR.  Following  is  some  information  to  help  with  interpreting  the  reliability  and 
validity  information  in  the  ROMR. 


What  Is  Reliability? 

Reliability  is  defined  as  the  consistency  or  dependability  of  a  measurement  procedure  or  the 
extent  to  which  a  measured  variable  is  free  from  random  error  (Leary,  1995;  Stangor,  2004; 
Pelham  and  Blanton,  2007). 

There  are  several  types  of  reliability,  including: 

•  Inter-rater  reliability:  used  to  assess  the  degree  to  which  different  raters/observers  give 
consistent  estimates  of  the  same  phenomenon 

•  Test-retest  reliability:  used  to  assess  the  consistency  of  a  measure  from  one  time  to 
another 

•  Internal  consistency  reliability:  used  to  assess  the  consistency  of  results  across  items 
within  a  test  (Rousson,  Gasser,  and  Seifert,  2002). 

Several  types  of  statistics  are  used  to  assess  reliability  including  correlations,  Cronbach’s 
alpha,  and  Cohen’s  kappa.  Reliability  statistics  are  generally  reported  somewhere  between  0.00 
(no  reliability)  and  1.00  (perfect  reliability).  Most  of  the  time,  if  two  measures  are  compared 
side  by  side,  the  one  with  the  reliability  statistic  closest  to  1.00  is  the  more  reliable  measure.  It 
is  important  to  note  that  the  number  of  items  in  a  measure  can  influence  its  reliability.  Users 
should  consider  both  the  number  of  items  and  reliability  information  when  making  a  decision 
about  which  measure  is  most  appropriate  (Leary,  1995;  Stangor,  2004;  Pelham  and  Blanton, 
2007;  Rousson,  Gasser,  and  Seifert,  2002). 
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What  Is  Validity? 

Validity  is  defined  as  the  extent  to  which  a  measurement  procedure  actually  measures  what  it 
is  intended  to  measure  (Leary,  1995). 

There  are  several  types  of  validity,  including: 

•  Criterion  validity:  used  to  assess  the  extent  to  which  the  measure  actually  captures  the 
phenomenon  it  is  trying  to  measure  (i.e.,  the  criterion)  by  comparing  (or  correlating)  it 
with  other  measures  of  the  same  construct 

•  Content  validity:  used  to  assess  the  extent  to  which  the  test  content  covers  a  representa¬ 
tive  sample  of  the  behavior  domain  to  be  measured 

•  Predictive  validity:  used  to  assess  the  extent  to  which  the  measure  can  predict  (or  cor¬ 
relate  with)  other  measures  of  the  same  construct  that  are  measured  at  some  time  in  the 
future  (Leary,  1995;  Rousson,  Gasser,  and  Seifert,  2002). 

Statistics  can  be  used  as  an  indicator  of  validity.  For  example,  to  establish  criterion  valid¬ 
ity,  a  measure  would  be  correlated  with  another  measure  of  the  same  construct  and  the  correla¬ 
tion  coefficient  would  represent  the  measure’s  criterion  validity.  Similar  to  reliability  statistics, 
the  correlation  can  range  from  0.00  (no  correlation)  to  1.00  (perfect  correlation),  with  higher 
correlations  indicating  a  higher  level  of  criterion  validity  (Leary,  1995).  It  is  important  to 
note  the  sample  size  in  the  article’s  publishing  information  on  validity  of  the  measure.  Larger 
sample  size  generally  provides  more  confidence  that  the  measure  has  been  validated  using  a 
representative  sample  of  the  target  population.  Users  should  consider  both  the  sample  size  and 
validity  information  when  making  a  decision  about  which  measure  is  most  appropriate. 


APPENDIX  H 


Brief  User  Guide  for  the  RAND  Online  Measure  Repository 


Getting  to  the  ROMR 

How  do  I  get  to  the  ROMR? 

The  ROMR  is  available  at  http://militaryhealth.rand.org/innovative-practices/measures.html 

Do  I  need  a  password  or  to  register  to  use  the  ROMR? 

No,  the  ROMR  is  free  for  anyone  to  use. 


Searching  for  Measures 

What  filters  can  I  use  to  help  find  the  measures  I'm  looking  for? 

The  ROMR  filters  measures  by 

•  Age 

•  Specific  clinical  domains  such  as  depression,  PTSD,  or  TBI 

•  Previous  use  with  a  military  population 

•  Cost  of  purchasing  the  measure 

•  Number  of  items  in  the  measure 

•  Whether  the  measure  is  intended  to  be  self-administered  or  must  be  administered  by  a 
clinician  or  trained  professional 

•  Respondent  (e.g.,  an  individual  or  caregiver  responding  about  their  child). 

Can  I  search  for  a  specific  measure  by  name  or  a  specific  keyword? 

Yes,  the  ROMR  allows  you  to  search  by  keyword  or  name  using  a  free  text  field.  Enter  the 
name  of  the  measure  or  keyword  into  the  “Keyword”  search  box  and  click  on  the  “Go”  button. 


Using  Search  Results 

Can  I  compare  measures  side  by  side? 

Yes,  to  compare  measures  side  by  side,  click  the  check  box  next  to  the  measures  you  want  to 
compare  and  then  click  on  the  “Compare  selected”  button. 
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Is  there  a  glossary  to  help  me  understand  what  is  meant  by  certain  words  or  domains 
included  in  the  ROMR? 

Yes,  a  glossary  that  defines  the  key  terms  used  in  the  ROMR  is  available  online  at 
http://militaryhealth.rand.org/innovative-practices/measures.html 

Some  of  the  search  results  get  cut  off  midsentence.  How  do  I  see  the  full  text? 

Clicking  on  the  text  that  says  “Read  More”  will  show  the  full  text. 
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Since  2001,  U.S.  military  forces  have  been  engaged  in  extended  conflicts  in  Iraq  and  Afghanistan.  While  most 
military  personnel  cope  well  across  the  deployment  cycle,  the  operational  tempo  may  raise  the  risk  of  mental  health 
problems,  such  as  post-traumatic  stress  disorder  (PTSD)  and  major  depression,  and  consequences  from  traumatic 
brain  injury  (TBI).  To  support  servicemembers  and  their  families  as  they  cope  with  these  challenges,  the  U.S. 
Department  of  Defense  has  implemented  numerous  programs  addressing  biological,  social,  spiritual,  and  holistic 
influences  on  psychological  health  along  the  resilience,  prevention,  and  treatment  continuum  that  focus  on  a  variety 
of  clinical  and  nonclinical  concerns. 

As  these  efforts  have  proliferated,  evaluating  their  effectiveness  has  become  increasingly  important.  To  support 
the  design  and  implementation  of  program  evaluation,  RAND  developed  the  RAND  Online  Measure  Repository 
(ROMR)  which  indexes  and  describes  measures  related  to  psychological  health  and  TBI.  The  ROMR  is  a  publicly 
accessible,  online,  searchable  database  containing  171  measures  related  to  psychological  health  and  TBI.  This 
report  describes  the  rationale  for  developing  the  ROMR,  the  content  included  in  the  ROMR,  and  its  potential  in  both 
civilian  and  military  populations.  The  ROMR  includes  information  about  measure  domains,  psychometrics,  number 
of  items,  and  costs,  which  can  inform  the  selection  of  measures  for  program  evaluations.  Included  measures  address 
domains  of  primary  importance  to  psychological  health  (PTSD,  depression,  anxiety,  suicidal  ideation,  and  resiliency) 
and  TBI  (cognition,  executive  functioning,  and  memory).  Also  identified  are  measures  relevant  to  military  units,  such 
as  unit  cohesion  and  force  readiness  and  preservation. 
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